mlsanityData to 7 Oct 2026

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

arXiv 2502.09560 ↗First posted 13 Feb 2025cs.AI cs.CL cs.CVcited by 213

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Leveraging Multi-modal Large Language Models (MLLMs) to create embodied agents offers a promising avenue for tackling real-world tasks. While language-centric embodied agents have garnered substantial attention, MLLM-based embodied agents remain underexplored due to the lack of comprehensive evaluation frameworks. To bridge this gap, we introduce EmbodiedBench, an extensive benchmark designed to evaluate vision-driven embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing tasks across four environments, ranging from high-level semantic tasks (e.g., household) to low-level tasks involving atomic actions (e.g., navigation and manipulation); and (2) six meticulously curated subsets evaluating essential agent capabilities like commonsense reasoning, complex instruction understanding, spatial awareness, visual perception, and long-term planning. Through extensive experiments, we evaluated 24 leading proprietary and open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only 28.9% on average. EmbodiedBench provides a multifaceted standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance MLLM-based embodied agents. Our code and dataset are available at https://embodiedbench.github.io.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 135 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

74 of 135 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents21 Jan
  2. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons11 Dec
  3. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents12 Aug
  4. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
  5. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  6. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  8. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  9. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 65

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  13. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  14. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  15. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  16. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  17. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  18. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  19. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  20. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  21. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  22. 2024RT-H: Action Hierarchies Using Languagecited by 204
  23. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  24. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  25. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  26. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  27. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  28. 2023Video Language Planningcited by 160
  29. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  30. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  31. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  32. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  33. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  34. 2024Large Multimodal Agents: A Surveycited by 99
  35. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  36. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  37. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Taskscited by 78
  38. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 84; 2026: 129.

213 papers.

Sort:

By area:

  1. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  2. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  3. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  4. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  6. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  7. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  8. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  9. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  10. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  11. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  12. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
38 more
  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  2. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  3. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  4. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  5. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  6. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 27
  7. 2025A Survey of Personalization: From RAG to Agentcited by 26
  8. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  9. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  10. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  11. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  12. 2025Embodied AI: From LLMs to World Modelscited by 23
  13. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  14. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21
  15. 2026ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loopcited by 19
  16. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  17. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  18. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  19. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14
  20. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 14
  21. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interactioncited by 14
  22. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 14
  23. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  24. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  25. 2025Reinforced Reasoning for Embodied Planningcited by 12
  26. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  27. 2026EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agentscited by 12
  28. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 11
  29. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  30. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  31. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  32. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
  33. 2025Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesiscited by 10
  34. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  35. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  36. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligencecited by 10
  37. 2025Pelican-VL 1.0: A Foundation Brain Model for Embodied Intelligencecited by 10
  38. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Plannercited by 9

50 most cited of 213 All 213 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BLM1: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning
  2. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
  3. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond
  4. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents
  5. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  6. 2026ST-BiBench: Benchmarking Multi-Stream Multimodal Coordination in Bimanual Embodied Tasks for MLLMs
  7. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agents
  8. 2025Igniting VLMs toward the Embodied Space