Looking at a component is an intuitive way to indicate interest, but turning that gaze into a robot movement requires geometry, calibration and planning. This working paper, co-authored with Alireza Rastegarpanah and Rustam Stolkin, investigates eye tracking for selecting targets and specifying paths in a mixed-reality battery-disassembly workcell.
From an Eye Direction to a Physical Point
An eye tracker estimates a direction of gaze. A direction alone does not identify a unique point: many possible surfaces lie along the same ray. Depth supplies the missing distance, allowing the system to associate the gaze with a location in the scene.
That location must then be expressed in coordinates the robot understands. Camera coordinates, the Unity workcell and the robot base are not automatically interchangeable. Calibration and coordinate transforms connect them; errors at those stages can remain even when the operator looks precisely at the desired feature.
The system uses a FOVE headset to present stereo images and depth information from a real camera through a Unity representation. Gaze-derived coordinates pass through ROS to MoveIt for task-specific trajectory planning.
graph TD
A[Eye-tracked gaze direction] --> C[Scene target point]
B[Stereo camera and depth] --> C
C --> D[Coordinate transformation into robot frame]
D --> E[MoveIt task planning]
E --> F[Simulated robot trajectory]
F --> G[Mixed-reality visual feedback]
G --> A
Original explanatory diagram showing why gaze estimation, depth and robot planning are distinct stages; it is not a reproduced paper figure.
A Target Is Not Yet a Valid Movement
Even a correctly reconstructed point may be unreachable or lie close to an obstacle. MoveIt must account for robot geometry and the task, rather than treating a gaze point as immediate permission to move.
The precision requirement also changes with the operation. Grasping may admit a useful region on an object. Unbolting requires alignment with a smaller feature. Cutting requires maintaining a path across many points, so an acceptable starting point does not guarantee an acceptable complete motion.
These differences help explain why an interface can work well for one task and poorly for another. “Gaze control accuracy” is not one universal number independent of target geometry and the definition of success.
Simulation Results
The evaluation contains 50 trials for each of three tasks.
| Task | Reported success rate | Successful trials out of 50 |
|---|---|---|
| Unbolting | 66% | 33 |
| Grasping | 94% | 47 |
| Cutting | 50% | 25 |
Mean unbolting target errors ranged from 5.59 to 8.78 millimetres. Mean cutting-path deviations were 4.03 millimetres horizontally and 3.03 millimetres vertically.
The relatively high grasp success should not be transferred to cutting or precision fastener alignment. Success criteria differ, and one additional successful attempt changes a 50-trial rate by two percentage points. The percentages describe the tested conditions, not an unlimited population of robot tasks.
Sources of Error and Interpretation
Depth uncertainty, dispersed gaze estimates and planning sensitivity limit the resulting motion. Improving eye tracking alone cannot remove an incorrect camera-to-robot transform or inaccurate depth. The whole geometric chain needs to be evaluated.
The experiment supports gaze-based target and path specification in simulation. Although the interface uses real camera information, that does not convert the manipulation evaluation into a safety-certified physical disassembly trial.
Potential accessibility benefits likewise need direct study with intended users. An interface that reduces manual pointing may still require confirmation mechanisms, tolerance to involuntary gaze shifts and a clear way to cancel a mistaken selection. These are design considerations, not features claimed as validated by the working paper.
Reproduction Boundaries
A faithful reproduction would require headset calibration, stereo/depth configuration, frame transforms, robot and tool models, planner settings, target geometry and the original task-success definitions. Without those, a similar-looking demonstration tests the interaction principle rather than reproducing the reported rates.
The useful conceptual result is the separation of intention input from motion validation: gaze proposes where or along which path to work, while the robot’s geometric model determines whether the proposal can be executed.
Testing and Real-World Use
This could be tested in a virtual workcell by comparing gaze-selected points and paths with known targets, measuring selection errors and task success separately for grasping, unbolting and cutting. In remote or assistive interfaces, gaze could reduce manual pointing effort while a confirmation and planning stage checks the intended target before motion.
Working Paper
The working paper dated 1 December 2023 contains the original system illustrations and detailed results. It is distinct from the Falling Ball thesis, which studies human interception rather than robot target selection.