# Gaze-based, Context-aware Robotic System for Assisted Reaching and Grasping

arXiv: https://arxiv.org/abs/1809.08095 · HTML: https://mlsanity.com/paper/1809.08095/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2018.

## Built on by (most cited first, 17 of 17)

- [Human Attention in Fine-grained Classification](https://mlsanity.com/paper/2111.01628/) (2021)
- [Intention estimation from gaze and motion features for human-robot shared-control object manipulation](https://mlsanity.com/paper/2208.08688/) (2022)
- [Gaze-based intention estimation: principles, methodologies, and applications in HRI](https://mlsanity.com/paper/2302.04530/) (2023)
- [Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding](https://mlsanity.com/paper/2510.21356/) (2025)
- [Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation](https://mlsanity.com/paper/2605.30282/) (2026)
- [Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention](https://mlsanity.com/paper/2404.07347/) (2024)
- [MIDAS: Deep learning human action intention prediction from natural eye movement patterns](https://mlsanity.com/paper/2201.09135/) (2022)
- [Non-invasive Cognitive-level Human Interfacing for the Robotic Restoration of Reaching & Grasping](https://mlsanity.com/paper/2102.12980/) (2021)
- [Gaze-contingent decoding of human navigation intention on an autonomous wheelchair platform](https://mlsanity.com/paper/2103.03072/) (2021)
- [I am Robot: Neuromuscular Reinforcement Learning to Actuate Human Limbs through Functional Electrical Stimulation](https://mlsanity.com/paper/2103.05349/) (2021)
- [Towards AI-controlled FES-restoration of arm movements: neuromechanics-based reinforcement learning for 3-D reaching](https://mlsanity.com/paper/2301.04004/) (2023)
- [Learning Multimodal AI Algorithms for Amplifying Limited User Input into High-dimensional Control Space](https://mlsanity.com/paper/2505.11366/) (2025)
- [Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach](https://mlsanity.com/paper/2512.16456/) (2025)
- [Towards AI-controlled FES-restoration of movements: Learning cycling stimulation pattern with reinforcement learning](https://mlsanity.com/paper/2303.09986/) (2023)
- [LocalEyenet: Deep Attention framework for Localization of Eyes](https://mlsanity.com/paper/2303.12728/) (2023)
- [Exploring AI-enhanced Shared Control for an Assistive Robotic Arm](https://mlsanity.com/paper/2306.13509/) (2023)
- [Gaze-Regularized VLMs for Ego-Centric Behavior Understanding](https://mlsanity.com/paper/2603.23190/) (2026)

## Abstract (arXiv)

Assistive robotic systems endeavour to support those with movement disabilities, enabling them to move again and regain functionality. Main issue with these systems is the complexity of their low-level control, and how to translate this to simpler, higher level commands that are easy and intuitive for a human user to interact with. We have created a multi-modal system, consisting of different sensing, decision making and actuating modalities, leading to intuitive, human-in-the-loop assistive robotics. The system takes its cue from the user's gaze, to decode their intentions and implement low-level motion actions to achieve high-level tasks. This results in the user simply having to look at the objects of interest, for the robotic system to assist them in reaching for those objects, grasping them, and using them to interact with other objects. We present our method for 3D gaze estimation, and grammars-based implementation of sequences of action with the robotic system. The 3D gaze estimation is evaluated with 8 subjects, showing an overall accuracy of 4.68±0.14cm. The full system is tested with 5 subjects, showing successful implementation of 100% of reach to gaze point actions and full implementation of pick and place tasks in 96%, and pick and pour tasks in 76% of cases. Finally we present a discussion on our results and what future work is needed to improve the system.