# In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting

arXiv: https://arxiv.org/abs/2509.07447 · HTML: https://mlsanity.com/paper/2509.07447/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 28 of 28)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition](https://mlsanity.com/paper/1803.09125/) (2018)
- [In the Eye of the Beholder: Gaze and Actions in First Person Video](https://mlsanity.com/paper/2006.00626/) (2020)
- [EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models](https://mlsanity.com/paper/2311.15596/) (2023)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering](https://mlsanity.com/paper/2502.07411/) (2025)
- [AMEGO: Active Memory from long EGOcentric videos](https://mlsanity.com/paper/2409.10917/) (2024)
- [MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA](https://mlsanity.com/paper/2410.07177/) (2024)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs](https://mlsanity.com/paper/2501.04336/) (2025)
- [GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear](https://mlsanity.com/paper/2401.17217/) (2024)

## Built on by (most cited first, 22 of 22)

- [StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos](https://mlsanity.com/paper/2512.01707/) (2025)
- [EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding](https://mlsanity.com/paper/2606.24422/) (2026)
- [EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning](https://mlsanity.com/paper/2511.18242/) (2025)
- [SkillSight: Efficient First-Person Skill Assessment with Gaze](https://mlsanity.com/paper/2511.19629/) (2025)
- [Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI](https://mlsanity.com/paper/2603.01104/) (2026)
- [Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach](https://mlsanity.com/paper/2512.16456/) (2025)
- [Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2601.06750/) (2026)
- [ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation](https://mlsanity.com/paper/2602.05132/) (2026)
- [EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next](https://mlsanity.com/paper/2603.12147/) (2026)
- [Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering](https://mlsanity.com/paper/2603.12533/) (2026)
- [GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions](https://mlsanity.com/paper/2605.15764/) (2026)
- [EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning](https://mlsanity.com/paper/2605.17262/) (2026)
- [Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models](https://mlsanity.com/paper/2606.09142/) (2026)
- [GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding](https://mlsanity.com/paper/2603.25841/) (2026)
- [From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants](https://mlsanity.com/paper/2604.08062/) (2026)
- [EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment](https://mlsanity.com/paper/2604.08342/) (2026)
- [VEGAS: Human-Aligned Video Caption Evaluation via Gaze](https://mlsanity.com/paper/2607.08489/) (2026)
- [EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input](https://mlsanity.com/paper/2608.15614/) (2026)
- [G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding](https://mlsanity.com/paper/2608.20157/) (2026)
- [Companion-style QA Assistance in Ego-Vision](https://mlsanity.com/paper/2609.06721/) (2026)
- [CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video](https://mlsanity.com/paper/2609.17688/) (2026)
- [GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?](https://mlsanity.com/paper/2610.15129/) (2026)

## Abstract (arXiv)

The emergence of advanced multimodal large language models (MLLMs) has significantly enhanced AI assistants' ability to process complex information across modalities. Recently, egocentric videos, by directly capturing user focus, actions, and context in an unified coordinate, offer an exciting opportunity to enable proactive and personalized AI user experiences with MLLMs. However, existing benchmarks overlook the crucial role of gaze as an indicator of user intent. To address this gap, we introduce EgoGazeVQA, an egocentric gaze-guided video question answering benchmark that leverages gaze information to improve the understanding of longer daily-life videos. EgoGazeVQA consists of gaze-based QA pairs generated by MLLMs and refined by human annotators. Our experiments reveal that existing MLLMs struggle to accurately interpret user intentions. In contrast, our gaze-guided intent prompting methods significantly enhance performance by integrating spatial, temporal, and intent-related cues. We further conduct experiments on gaze-related fine-tuning and analyze how gaze estimation accuracy impacts prompting effectiveness. These results underscore the value of gaze for more personalized and effective AI assistants in egocentric settings. Project page: https://taiyi98.github.io/projects/EgoGazeVQA