Ego-Grounding for Personalized Question-Answering in Egocentric Videos
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 56 references that are in the map. First posted on arXiv 2 Apr 2026.
Builds on
50 of 56 references.
Closest ancestors · 3
Same direction, newest first
- 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges17 Nov
- 2025EgoBlind: Towards Egocentric Visual Assistance for the Blind11 Mar
- 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models27 Nov
Other references · 47
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
35 more
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
- 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
- 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
- 2024Long Context Transfer from Language to Visioncited by 502
- 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
- 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
- 2022Egocentric Video-Language Pretrainingcited by 231
- 2022Learning Video Representations from Large Language Modelscited by 204
- 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
- 2024Streaming Long Video Understanding with Large Language Modelscited by 164
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
- 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
- 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
- 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
- 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
- 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
- 2023An Outlook into the Future of Egocentric Visioncited by 84
- 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
- 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
- 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
- 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
- 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
- 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
- 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
- 2021Self-Regulated Learning for Egocentric Video Activity Anticipationcited by 32
- 2024Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mindcited by 28
- 2023RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4Dcited by 21
- 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Videocited by 16
- 2025The CASTLE 2024 Dataset: Advancing the Art of Multimodal Understandingcited by 15
- 2023EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understandingcited by 14
- 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
- 2025Visual Intention Grounding for Egocentric Assistantscited by 11
- 2025TeleEgo: Benchmarking Egocentric AI Assistants in the Wildcited by 9
- 2025Mitigating Easy Option Bias in Multiple-Choice Question Answeringcited by 4
- 2025Causal-Entity Reflected Egocentric Traffic Accident Video Synthesiscited by 2
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026LightMem-Ego: Your AI Memory for Everyday Lifecited by 2
- 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memorycited by 2
- 2026EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenescited by 0
- 2026Companion-style QA Assistance in Ego-Visioncited by 0
- 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0
- 2026EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videoscited by 0
- 2026EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos30 Sep 2026
- 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization17 Sep 2026
- 2026Companion-style QA Assistance in Ego-Vision6 Sep 2026
- 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory13 Aug 2026
- 2026LightMem-Ego: Your AI Memory for Everyday Life13 Jul 2026
- 2026EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes1 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering
- 2025Advancing Egocentric Video Question Answering with Multimodal Large Language Models
- 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
- 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
- 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA
- 2026The Right Inference Strategy Is All You Need: Nearly Training-Free Domain-Wise Inference for EgoCross Challenge
- 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos
- 2025Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance