IMBench: A Benchmark for Intuitive Robotic Manipulation
On the map: Vision-language-action models → Failure recovery and robustness
Abstract (arXiv)
Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 70 of its 87 references that are in the map. First posted on arXiv 17 Jul 2026.
Builds on
70 of 87 references.
References · 70
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
38 more
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
- 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
- 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
- 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
- 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
- 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
- 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
- 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
- 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
- 2020Integrated Task and Motion Planningcited by 281
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
- 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
- 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
- 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
- 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
- 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
- 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
- 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
- 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
- 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
- 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
- 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
- 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
- 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
- 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
- 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
- 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
- 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
- 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
- 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
- 2019CoPhy: Counterfactual Learning of Physical Dynamicscited by 80
- 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
- 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 61
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
- 2026CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
- 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation
- 2026RoboHarn-Evo: Evolving Hierarchical Physical Knowledge for Self-Improving Robotic Manipulation
- 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
- 2026DuoBench: A Reproducible Benchmark for Bimanual Manipulation in Simulation and the Real World
- 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
- 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation