Solving Sensor Occlusion in the IR Array

The fundamental flaw with most optical hand-tracking systems (like those built into VR headsets) is that they rely on RGB cameras placed far away from the subject, relying heavily on a beefy CPU to infer skeletal structure from a noisy image.

When building BoardHand, we had the opposite problem. The cameras are incredibly close to the subject (the hands), and they look up. This provides excellent resolution for the bottom of the hand, but terrible visibility for the fingertips if the palm obscures the view. We call this the occlusion problem.

The Three-Sensor Solution

Our initial prototype used two sensors at the extreme top corners of the board. This created a wide stereoscopic baseline, but left a massive blind spot directly in the center of the board—right where the index fingers do most of their work.

Moving to a three-sensor array (Left, Center, Right) solved the blind spot, but introduced a compute problem. Processing three simultaneous video feeds at 120Hz required too much bandwidth for our target ARM Cortex-M7 MCU.

Predictive Kinematics

The breakthrough came when we realized we didn't need to track the whole hand at 120Hz. The human hand has physical constraints. Fingers cannot teleport.

We implemented a lightweight Kalman filter tailored specifically to human biomechanics. If the Left sensor loses sight of the index fingertip because the thumb blocks it, the MCU calculates the fingertip's last known velocity and trajectory, while confirming the hand's overall position via the Center sensor. It "guesses" where the tip is for the next 15-20 milliseconds until it reappears.

To the user, the cursor never drops, stutters, or jumps. The math fills in the physical gaps.