mlsanityData to 14 Oct 2026

EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents

arXiv 2501.11858 ↗First posted 21 Jan 2025cs.CV cs.CLcited by 57

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied AI benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied 3D scenes, each of which is rigorously selected and annotated. It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity, all within a unified simulation and evaluation framework tailored for MLLMs. The tasks are organized into five categories: navigation, object interaction, social interaction, attribute question answering, and spatial question answering to assess different capabilities of the agents. We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. We open-source all evaluation data and simulation framework at https://github.com/thunlp/EmbodiedEval.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 71 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

50 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024LEGENT: Open Platform for Embodied Agents28 Apr
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning11 Dec
  5. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 44

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
32 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  4. 2018On Evaluation of Embodied Navigation Agentscited by 547
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  6. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  8. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  9. 2017Embodied Question Answeringcited by 336
  10. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  11. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  12. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  13. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  14. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
  15. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  16. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  17. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  18. 2020Rearrangement: A Challenge for Embodied AIcited by 155
  19. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  20. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  21. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  22. 2021Visual Room Rearrangementcited by 102
  23. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  24. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 94
  25. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  26. 2019Multi-Target Embodied Question Answeringcited by 74
  27. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  28. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  29. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  30. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  31. 2024LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environmentscited by 6
  32. 2024DOZE: A Dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environmentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 28.

57 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  4. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  5. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  6. 2026ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loopcited by 26
  7. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  8. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  9. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 17
  10. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agentscited by 15
  11. 2025PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understandingcited by 14
  12. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
38 more
  1. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  2. 2025Safety of Embodied Navigation: A Surveycited by 10
  3. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 10
  4. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  5. 2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 9
  6. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  7. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 7
  8. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  9. 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Taskscited by 7
  10. 2025EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Abilitycited by 6
  11. 2025Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Huntingcited by 6
  12. 2025DeliveryBench: Can Agents Earn Profit in Real World?cited by 6
  13. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  14. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agentscited by 5
  15. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Makingcited by 4
  16. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  17. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  18. 2026From Perception to Action: An Interactive Benchmark for Vision Reasoningcited by 4
  19. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 4
  20. 2026Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videoscited by 4
  21. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  22. 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratoriescited by 3
  23. 2026PhotoCraft: Agentic Reasoning with Hierarchical Self-Evolving Memory for Deep Image Searchcited by 3
  24. 2026AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agentscited by 3
  25. 2025Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligencecited by 2
  26. 2025ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learningcited by 2
  27. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspectivecited by 2
  28. 2025Static and Plugged: Make Embodied Evaluation Simplecited by 1
  29. 2025Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environmentscited by 1
  30. 2025OceanGym: A Benchmark Environment for Underwater Embodied Agentscited by 1
  31. 2026ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulationcited by 1
  32. 2026IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instructioncited by 1
  33. 2026Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trendscited by 1
  34. 2026RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodimentscited by 1
  35. 2025Evaluating point-light biological motion in multimodal large language modelscited by 0
  36. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agentscited by 0
  37. 2026MirrorBench: Evaluating Self-centric Intelligence in MLLMs by Introducing a Mirrorcited by 0
  38. 2026A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluationcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Grounding Multimodal Large Language Models in Actions
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  3. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
  6. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  7. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  8. 2026What We are Missing in Multimodal LLM Evaluation?