mlsanityData to 14 Oct 2026

EgoLife: Towards Egocentric Life Assistant

arXiv 2503.03803 ↗First posted 5 Mar 2025cs.CVcited by 91

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Builds on 117 of its 176 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

117 of 176 references.

References · 117

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  12. 2017Dense-Captioning Events in Videoscited by 793
38 more
  1. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  2. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  3. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  5. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  6. 2024Qwen2-Audio Technical Reportcited by 560
  7. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  8. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  9. 2024Long Context Transfer from Language to Visioncited by 515
  10. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  11. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  12. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  13. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  14. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  15. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  16. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  17. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  18. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  19. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  20. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  21. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  22. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  23. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  24. 2022Egocentric Video-Language Pretrainingcited by 232
  25. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  26. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  27. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  28. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  29. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  30. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 167
  31. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  32. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  33. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  34. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  35. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  36. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  37. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 139
  38. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134

50 most cited of 117 All 117 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 69.

91 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  3. 2026A Simple Baseline for Streaming Video Understandingcited by 33
  4. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 29
  5. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  6. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 23
  7. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  8. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 22
  9. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 21
  10. 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answeringcited by 21
  11. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 19
  12. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
38 more
  1. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  2. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  3. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  4. 2025Developing Vision-Language-Action Model from Egocentric Videoscited by 11
  5. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  6. 2025Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agentscited by 8
  7. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  8. 2025Target-Aware Video Diffusion Modelscited by 7
  9. 2025VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotationscited by 7
  10. 2026What Should a Streaming Video Model Remember?cited by 7
  11. 2025EgoTrigger: Toward Audio-Driven Image Capture for Human Memory Enhancement in All-Day Energy-Efficient Smart Glassescited by 6
  12. 2026VisionClaw: Always-On AI Agents through Smart Glassescited by 6
  13. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 6
  14. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 6
  15. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  16. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  17. 2026OmniEgo-R2: A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026cited by 5
  18. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 5
  19. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
  20. 2026HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editingcited by 5
  21. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 5
  22. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  23. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  24. 2026Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsingcited by 4
  25. 2026Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoningcited by 4
  26. 2026EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchycited by 4
  27. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learningcited by 3
  28. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 3
  29. 2026Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrievalcited by 3
  30. 2026SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memorycited by 3
  31. 2026ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Enginecited by 3
  32. 2026StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understandingcited by 3
  33. 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memorycited by 3
  34. 2026EgoSelf: From Memory to Personalized Egocentric Assistantcited by 2
  35. 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Controlcited by 2
  36. 2026M3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Taskscited by 2
  37. 2026S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrievalcited by 2
  38. 2026Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understandingcited by 2

50 most cited of 91 All 91 (JSON)