# Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

arXiv: https://arxiv.org/abs/2510.17801 · HTML: https://mlsanity.com/paper/2510.17801/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 63 of its 71 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 63)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [MolmoAct: Action Reasoning Models that can Reason in Space](https://mlsanity.com/paper/2508.07917/) (2025)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model](https://mlsanity.com/paper/2503.10631/) (2025)
- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)

## Built on by (most cited first, 25 of 25)

- [HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents](https://mlsanity.com/paper/2604.07430/) (2026)
- [Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test](https://mlsanity.com/paper/2601.04137/) (2026)
- [Hy-Embodied-VLM-1.0: Efficient Physical-World Agents](https://mlsanity.com/paper/2607.12894/) (2026)
- [Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training](https://mlsanity.com/paper/2512.24125/) (2025)
- [RoboAgent: Chaining Basic Capabilities for Embodied Task Planning](https://mlsanity.com/paper/2604.07774/) (2026)
- [OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2606.03890/) (2026)
- [MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents](https://mlsanity.com/paper/2511.23055/) (2025)
- [FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies](https://mlsanity.com/paper/2605.27284/) (2026)
- [Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models](https://mlsanity.com/paper/2603.15618/) (2026)
- [LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories](https://mlsanity.com/paper/2603.11987/) (2026)
- [HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks](https://mlsanity.com/paper/2603.12760/) (2026)
- [From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligence](https://mlsanity.com/paper/2607.11689/) (2026)
- [CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding](https://mlsanity.com/paper/2608.30289/) (2026)
- [ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination](https://mlsanity.com/paper/2609.24526/) (2026)
- [Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents](https://mlsanity.com/paper/2511.18685/) (2025)
- [Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration](https://mlsanity.com/paper/2605.31196/) (2026)
- [A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation](https://mlsanity.com/paper/2606.10366/) (2026)
- [RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation](https://mlsanity.com/paper/2607.18709/) (2026)
- [FailBench: How Reliable are VLMs at Judging Robot Task Success?](https://mlsanity.com/paper/2609.03611/) (2026)
- [EmbodiedMind: Adaptive Data Curation and Prefix-Tree Reinforcement Learning for Efficient Embodied Intelligence](https://mlsanity.com/paper/2609.19659/) (2026)
- [CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation](https://mlsanity.com/paper/2609.33807/) (2026)
- [RoboFL: Federated Expert Assembly for World Action Models](https://mlsanity.com/paper/2609.34968/) (2026)
- [SkillWeaver: Agentic Exploration over Neural Interaction Skills for Scalable Robot Data Generation](https://mlsanity.com/paper/2609.36171/) (2026)
- [WOVEN: Weaving Visual World Modeling into Multimodal LLMs](https://mlsanity.com/paper/2610.12417/) (2026)
- [MC-EvoBench: Auditing Experience-Driven Self-Improvement in Minecraft Agents](https://mlsanity.com/paper/2610.13829/) (2026)

## Abstract (arXiv)

Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems often follow a dual-system paradigm, where System 2 performs high-level reasoning and System 1 handles low-level control. We refer to System 2 as the embodied brain, the cognitive core for decision-making in manipulation. Although evaluating this embodied brain is crucial, existing benchmarks mainly measure execution success or cover only limited aspects of high-level cognition and task realism. We introduce RoboBench, a benchmark for evaluating multimodal large language models (MLLMs) as embodied brains. RoboBench covers five dimensions: Instruction Comprehension, Perception Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis. It spans 14 capabilities, 25 tasks, and 6,092 QA pairs. To improve realism, it draws from large-scale real robotic data and in-house collection across diverse embodiments, attribute-rich objects, multi-view scenes, and memory-driven navigation. For planning, RoboBench introduces an MLLM-as-world-simulator framework that assesses whether predicted plans can achieve critical object-state changes under physical and visual constraints, enabling more faithful evaluation of long-horizon reasoning than symbolic matching. Experiments on 18 state-of-the-art MLLMs reveal persistent limitations in implicit instruction understanding, spatiotemporal reasoning, cross-scenario planning, fine-grained affordance understanding, and failure diagnosis. We further analyze how embodied cognitive abilities relate to downstream robotic control. RoboBench offers a comprehensive scaffold for quantifying high-level cognition and guiding next-generation MLLMs toward more robust robotic intelligence.