mlsanityData to 7 Oct 2026

Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding

arXiv 2501.00358 ↗First posted 31 Dec 2024cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

This paper investigates the problem of understanding dynamic 3D scenes from egocentric observations, a key challenge in robotics and embodied AI. Unlike prior studies that explored this as long-form video understanding and utilized egocentric video only, we instead propose an LLM-based agent, Embodied VideoAgent, which constructs scene memory from both egocentric video and embodied sensory inputs (e.g. depth and pose sensing). We further introduce a VLM-based approach to automatically update the memory when actions or activities over objects are perceived. Embodied VideoAgent attains significant advantages over counterparts in challenging reasoning and planning tasks in 3D scenes, achieving gains of 4.9% on Ego4D-VQ3D, 5.8% on OpenEQA, and 11.7% on EnvQA. We have also demonstrated its potential in various embodied AI tasks including generating embodied interactions and perception for robot manipulation. The code and demo will be made public.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 59 of its 69 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

59 of 69 references.

References · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
38 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  7. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  9. 2020Rescaling Egocentric Visioncited by 529
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  13. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  14. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  15. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  16. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  17. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  18. 2022Egocentric Video-Language Pretrainingcited by 231
  19. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  20. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  21. 2022Learning Video Representations from Large Language Modelscited by 204
  22. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  23. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  24. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  25. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  26. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  27. 2021Towards Long-Form Video Understandingcited by 146
  28. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  29. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  30. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  31. 2023MindAgent: Emergent Gaming Interactioncited by 87
  32. 2022Temporal Alignment Networks for Long-term Videocited by 84
  33. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  34. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  35. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  36. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  37. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  38. 2023EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understandingcited by 39

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 12.

24 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  3. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  4. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  5. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  6. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  7. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  8. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  9. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  10. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  11. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  12. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 3
12 more
  1. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  2. 2025P3: Toward Versatile Embodied Agentscited by 2
  3. 2025X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoningcited by 2
  4. 2026ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Explorationcited by 1
  5. 2026Agentic Collaborative Cognition for Zero-Shot 3D Understandingcited by 0
  6. 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 0
  7. 2026ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understandingcited by 0
  8. 2026Active Spatial Inspection for Effective and Efficient Embodied Explorationcited by 0
  9. 2026Agentic Video Understanding: A Surveycited by 0
  10. 2026Simple Agentic Memory for Generalist Robot Policiescited by 0
  11. 2026Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographiescited by 0
  12. 2026Uruqi: Learning Spatial Cognition from Visual Experiencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
  2. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agents
  3. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning
  4. 2026EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks
  5. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems
  6. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces
  7. 2025RynnEC: Bringing MLLMs into Embodied World
  8. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding