mlsanityData to 7 Oct 2026

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

arXiv 2410.07166 ↗First posted 9 Oct 2024cs.CL cs.AI cs.LGcited by 142

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 101 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

10 of 101 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents13 Feb

Other references · 9

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  4. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  5. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  6. 2024HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environmentscited by 32
  7. 2023Lifelong Robot Learning with Human Assisted Language Plannerscited by 19
  8. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 14
  9. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 66; 2026: 72.

142 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  3. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  5. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  6. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  7. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  8. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  9. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agentscited by 39
  10. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  11. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  12. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
38 more
  1. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  2. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  3. 2025AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answeringcited by 29
  4. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  5. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  6. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  7. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  8. 2025Position: Agent Should Invoke External Tools ONLY When Epistemically Necessarycited by 22
  9. 2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 22
  10. 2025DynTaskMAS: A Dynamic Task Graph-driven Framework for Asynchronous and Parallel LLM-based Multi-Agent Systemscited by 20
  11. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 19
  12. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  13. 2025Constrained Discrete Diffusioncited by 16
  14. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  15. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Themcited by 16
  16. 2025Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligencecited by 15
  17. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  18. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 13
  19. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 13
  20. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representationscited by 11
  21. 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 11
  22. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  23. 2024Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policycited by 9
  24. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 9
  25. 2025Characterizing LLM-driven Social Network: The Chirper.ai Casecited by 9
  26. 2025TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Taskscited by 9
  27. 2024LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulationcited by 8
  28. 2024Reflection-Bench: Evaluating Epistemic Agency in Large Language Modelscited by 7
  29. 2025REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?cited by 7
  30. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  31. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  32. 2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 7
  33. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  34. 2025HARBOR: Exploring Persona Dynamics in Multi-Agent Competitioncited by 6
  35. 2025BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Constructioncited by 6
  36. 2026Uncertainty-Aware Clarification in LLM Agents with Information Gaincited by 6
  37. 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Taskscited by 6
  38. 2026Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligencecited by 6

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond
  2. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  3. 2024Embodied CoT Distillation From LLM To Off-the-shelf Agents
  4. 2025Embodied AI: From LLMs to World Models
  5. 2026Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models
  7. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  8. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents