mlsanityData to 7 Oct 2026

ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation

arXiv 2505.09698 ↗First posted 14 May 2025cs.RO cs.AIcited by 24

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have revolutionized artificial intelligence and robotics due to their commonsense reasoning capabilities. In robotic manipulation, VLMs are used primarily as high-level planners, but recent work has also studied their lower-level reasoning ability, which refers to making decisions about precise robot movements. However, the community currently lacks a clear and common benchmark that can evaluate how well VLMs can aid low-level reasoning in robotics. Consequently, we propose a novel benchmark, ManipBench, to evaluate the low-level robot manipulation reasoning capabilities of VLMs across various dimensions, including how well they understand object-object interactions and deformable object manipulation. We extensively test 33 representative VLMs across 10 model families on our benchmark, including variants to test different model sizes. Our evaluation shows that the performance of VLMs significantly varies across tasks, and there is a strong correlation between this performance and trends in our real-world manipulation tasks. It also shows that there remains a significant gap between these models and human-level understanding. See our website at: https://manipbench.github.io.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 85 of its 101 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

85 of 101 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks24 Dec
  2. 2024Octopi: Object Property Reasoning with Large Tactile-Language Models5 May
  3. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  5. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?10 Oct

Other references · 80

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  13. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  14. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  15. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  16. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  17. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  18. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  19. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  20. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  21. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  22. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  23. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  24. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  25. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  26. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  27. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  28. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  29. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  30. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  31. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  32. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  33. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  34. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  35. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  36. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  37. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  38. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 13.

24 papers.

Sort:

By area:

  1. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  2. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  3. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  4. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  5. 2025HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interactioncited by 4
  6. 2025BOP-ASK: Object-Interaction Reasoning for Vision-Language Modelscited by 4
  7. 2025PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?cited by 3
  8. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  9. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Modelscited by 3
  10. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  11. 2025Long-Horizon Visual Imitation Learning via Plan and Code Reflectioncited by 1
  12. 2025TRACE: Textual Reasoning for Affordance Coordinate Extractioncited by 1
12 more
  1. 2025ResponsibleRobotBench: Benchmarking Responsible Robot Manipulation using Multi-modal Large Language Modelscited by 1
  2. 2026TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinementcited by 0
  3. 2026Revealing Interpretable Failure Modes of VLMscited by 0
  4. 2026RobotValues: Evaluating Household Robots When Human Values Conflictcited by 0
  5. 2026When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QAcited by 0
  6. 2026A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluationcited by 0
  7. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applicationscited by 0
  8. 2026Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Predictioncited by 0
  9. 2026GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failurescited by 0
  10. 2026From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvestingcited by 0
  11. 2026VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Controlcited by 0
  12. 2026ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  2. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  3. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots
  4. 2025Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulation
  5. 2026FailBench: How Reliable are VLMs at Judging Robot Task Success?
  6. 2024Task-oriented Robotic Manipulation with Vision Language Models
  7. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  8. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation