mlsanityData to 7 Oct 2026

In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting

arXiv 2509.07447 ↗First posted 9 Sep 2025cs.CVcited by 21

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The emergence of advanced multimodal large language models (MLLMs) has significantly enhanced AI assistants' ability to process complex information across modalities. Recently, egocentric videos, by directly capturing user focus, actions, and context in an unified coordinate, offer an exciting opportunity to enable proactive and personalized AI user experiences with MLLMs. However, existing benchmarks overlook the crucial role of gaze as an indicator of user intent. To address this gap, we introduce EgoGazeVQA, an egocentric gaze-guided video question answering benchmark that leverages gaze information to improve the understanding of longer daily-life videos. EgoGazeVQA consists of gaze-based QA pairs generated by MLLMs and refined by human annotators. Our experiments reveal that existing MLLMs struggle to accurately interpret user intentions. In contrast, our gaze-guided intent prompting methods significantly enhance performance by integrating spatial, temporal, and intent-related cues. We further conduct experiments on gaze-related fine-tuning and analyze how gaze estimation accuracy impacts prompting effectiveness. These results underscore the value of gaze for more personalized and effective AI assistants in egocentric settings. Project page: https://taiyi98.github.io/projects/EgoGazeVQA

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 40 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

28 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA9 Oct
  2. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models27 Nov

Other references · 26

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
14 more
  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  2. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  3. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  4. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  5. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  6. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  7. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 68
  8. 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
  9. 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
  10. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  11. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  12. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  13. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  14. 2024GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewearcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 17.

21 papers.

Sort:

By area:

  1. 2025StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videoscited by 8
  2. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  3. 2025SkillSight: Efficient First-Person Skill Assessment with Gazecited by 4
  4. 2026Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AIcited by 3
  5. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  6. 2025Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reachcited by 1
  7. 2026Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Modelscited by 1
  8. 2026ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimationcited by 1
  9. 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Nextcited by 1
  10. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answeringcited by 1
  11. 2026GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactionscited by 1
  12. 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoningcited by 1
9 more
  1. 2026GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understandingcited by 0
  2. 2026From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistantscited by 0
  3. 2026EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environmentcited by 0
  4. 2026Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Modelscited by 0
  5. 2026VEGAS: Human-Aligned Video Caption Evaluation via Gazecited by 0
  6. 2026EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Inputcited by 0
  7. 2026G3Ego: Gaze-Guided Graphs for Egocentric Action Understandingcited by 0
  8. 2026Companion-style QA Assistance in Ego-Visioncited by 0
  9. 2026CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Videocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
  2. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
  3. 2026Gaze-Regularized VLMs for Ego-Centric Behavior Understanding
  4. 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
  5. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs
  6. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding
  7. 2026EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports
  8. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videos