VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control
Not on the map. Why?
Abstract (arXiv)
Spatial intelligence requires more than describing object locations. Under incomplete observation, models must identify and acquire missing evidence, interpret it in a common spatial frame, and act on it. We introduce VA-Bench to evaluate the complete observe-reason-act-revise loop. General-purpose MLLMs learn procedural context from RGB-only demonstrations, actively select camera viewpoints, issue metric Cartesian commands, and revise them from execution feedback. Models receive no privileged object poses, oracle trajectories, or learned action heads. A fixed model-agnostic controller executes only model-specified targets. VA-Bench contains 14 base task families (11 single-arm and three dual-arm), seven held-out geometry/layout variants, and a long-horizon five-object composition track. We evaluate 12 primary model conditions in three independent runs over the same 20 physically verified seeds per base task, reporting terminal success, nine trajectory-level behavioral diagnostics, and subtask progress. First, the best-performing model scores 100.0% on target localization and 78.9% on spatial relations in the annotated run. Its three-run macro-average task success is only 53.93+/-3.17%. Second, active camera control significantly improves task success over passive multi-view observation. In one matched comparison, success rises from 27.86% to 57.50%. Third, held-out geometric transfer can reduce task success by over 30 percentage points. No model completes a strict long-horizon episode, despite substantial partial progress. VA-Bench thus tests whether general-purpose MLLMs can turn visual demonstrations and actively acquired evidence into successful embodied action.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 68 references that are in the map. First posted on arXiv 17 Sep 2026.
Builds on
59 of 68 references.
References · 59
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
38 more
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
- 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
- 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
- 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
- 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
- 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
- 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
- 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
- 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
- 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
- 2022Visual Spatial Reasoningcited by 288
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
- 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
- 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
- 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
- 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
- 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
- 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
- 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
- 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
- 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
- 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
- 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
- 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
- 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
- 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
- 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Modelscited by 54
- 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
50 most cited of 59 All 59 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026VCN-Bench: A Video-Contextualized Navigation Benchmark for Spatial Reasoning over Prior Visual Experience
- 2026SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
- 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models
- 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
- 2025Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning
- 2026From Reasoning Failures to Composable Video Spatial Intelligence
- 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence
- 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories