# Eye Gaze as a Signal for Conveying User Attention in Contextual AI Systems

arXiv: https://arxiv.org/abs/2501.13878 · HTML: https://mlsanity.com/paper/2501.13878/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 12 of 12)

- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges](https://mlsanity.com/paper/2311.03287/) (2023)
- [Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition](https://mlsanity.com/paper/1803.09125/) (2018)
- [GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear](https://mlsanity.com/paper/2401.17217/) (2024)
- [Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges](https://mlsanity.com/paper/2305.14080/) (2023)
- [G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios](https://mlsanity.com/paper/2405.07652/) (2024)
- [eXtended Artificial Intelligence: New Prospects of Human-AI Interaction Research](https://mlsanity.com/paper/2103.15004/) (2021)
- [Implicit gaze research for XR systems](https://mlsanity.com/paper/2405.13878/) (2024)

## Built on by (most cited first, 2 of 2)

- [Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Tasks](https://mlsanity.com/paper/2605.04227/) (2026)
- [GazeMind: A Gaze-Guided LLM Agent for Personalized Cognitive Load Assessment](https://mlsanity.com/paper/2605.05790/) (2026)

## Abstract (arXiv)

Advanced multimodal AI agents can now collaborate with users to solve challenges in the world. Yet, these emerging contextual AI systems rely on explicit communication channels between the user and system. We hypothesize that implicit communication of the user's interests and intent would reduce friction and improve user experience when collaborating with AI agents. In this work, we explore the potential of wearable eye tracking to convey signals about user attention. We measure the eye tracking signal quality requirements to effectively map gaze traces to physical objects, then conduct experiments that provide visual scanpath history as additional context when querying vision language models. Our results show that eye tracking provides high value as a user attention signal and can convey important context about the user's current task and interests, improving understanding of contextual AI agents.