mlsanityData to 14 Oct 2026

ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark

arXiv 2501.05031 ↗First posted 9 Jan 2025cs.CV cs.LG cs.ROcited by 19

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The enhancement of generalization in robots by large vision-language models (LVLMs) is increasingly evident. Therefore, the embodied cognitive abilities of LVLMs based on egocentric videos are of great interest. However, current datasets for embodied video question answering lack comprehensive and systematic evaluation frameworks. Critical embodied cognitive issues, such as robotic self-cognition, dynamic scene perception, and hallucination, are rarely addressed. To tackle these challenges, we propose ECBench, a high-quality benchmark designed to systematically evaluate the embodied cognitive abilities of LVLMs. ECBench features a diverse range of scene video sources, open and varied question formats, and 30 dimensions of embodied cognition. To ensure quality, balance, and high visual dependence, ECBench uses class-independent meticulous human annotation and multi-round question screening strategies. Additionally, we introduce ECEval, a comprehensive evaluation system that ensures the fairness and rationality of the indicators. Utilizing ECBench, we conduct extensive evaluations of proprietary, open-source, and task-specific LVLMs. ECBench is pivotal in advancing the embodied cognitive capabilities of LVLMs, laying a solid foundation for developing reliable core models for embodied agents. All data and code are available at https://github.com/Rh-Dang/ECBench.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 42 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

35 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain21 Feb
  2. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models16 Feb

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
21 more
  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  2. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  4. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  5. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  6. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  8. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  9. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  12. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
  13. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  14. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  15. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  16. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 45
  17. 2016MarioQA: Answering Questions by Watching Gameplay Videoscited by 42
  18. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  19. 2024Temporal Reasoning Transfer from Text to Videocited by 38
  20. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Modelscited by 36
  21. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understandingcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  2. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 36
  3. 2025Embodied AI: From LLMs to World Modelscited by 24
  4. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
  5. 2025CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoningcited by 13
  6. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 12
  7. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  8. 2025Multi-Modal Multi-Task (M3T) Federated Foundation Models for Embodied AI: Potentials and Challenges for Edge Integrationcited by 6
  9. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  10. 2026LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Taskscited by 5
  11. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 4
  12. 2026Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsingcited by 4
7 more
  1. 2026GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agentscited by 3
  2. 2025How Good are Foundation Models in Step-by-Step Embodied Reasoning?cited by 1
  3. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
  4. 2025Evaluating point-light biological motion in multimodal large language modelscited by 0
  5. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agentscited by 0
  6. 2026MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agentscited by 0
  7. 2026Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
  2. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
  3. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
  4. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
  5. 2026Extending Embodied Question Answering from Perception to Decision
  6. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents
  7. 2026Thinker: A vision-language foundation model for embodied intelligence
  8. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs