mlsanityData to 7 Oct 2026

VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks

arXiv 2412.18194 ↗First posted 24 Dec 2024cs.RO cs.AI cs.CLcited by 171

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

General-purposed embodied agents are designed to understand the users' natural instructions or intentions and act precisely to complete universal tasks. Recently, methods based on foundation models especially Vision-Language-Action models (VLAs) have shown a substantial potential to solve language-conditioned manipulation (LCM) tasks well. However, existing benchmarks do not adequately meet the needs of VLAs and relative algorithms. To better define such general-purpose tasks in the context of LLMs and advance the research in VLAs, we present VLABench, an open-source benchmark for evaluating universal LCM task learning. VLABench provides 100 carefully designed categories of tasks, with strong randomization in each category of task and a total of 2000+ objects. VLABench stands out from previous benchmarks in four key aspects: 1) tasks requiring world knowledge and common sense transfer, 2) natural language instructions with implicit human intentions rather than templates, 3) long-horizon tasks demanding multi-step reasoning, and 4) evaluation of both action policies and language model capabilities. The benchmark assesses multiple competencies including understanding of mesh&texture, spatial relationship, semantic instruction, physical laws, knowledge transfer and reasoning, etc. To support the downstream finetuning, we provide high-quality training data collected via an automated framework incorporating heuristic skills and prior information. The experimental results indicate that both the current state-of-the-art pretrained VLAs and the workflow based on VLMs face challenges in our tasks.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 65 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

45 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation3 Sep

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
32 more
  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  12. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  13. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  14. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  15. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  16. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  17. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  18. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  19. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  20. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  21. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  22. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  23. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  24. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  25. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  26. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  27. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  28. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  29. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  30. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  31. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  32. 2024RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarioscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 52; 2026: 119.

171 papers.

Sort:

By area:

  1. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  2. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  3. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  4. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  5. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 90
  6. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  7. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  8. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  9. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  10. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  11. 2026RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policiescited by 35
  12. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
38 more
  1. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  2. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 33
  3. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  4. 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluationcited by 31
  5. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  6. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  7. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  8. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 25
  9. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  10. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  11. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  12. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  13. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 21
  14. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  15. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  16. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 17
  17. 2026ActionCodec: What Makes for Good Action Tokenizerscited by 17
  18. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 17
  19. 2025RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulationcited by 16
  20. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 15
  21. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14
  22. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  23. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12
  24. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  25. 2025VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modelingcited by 11
  26. 2026Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibrationcited by 11
  27. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 10
  28. 2026World Action Models: A Surveycited by 10
  29. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 10
  30. 2026Long-Horizon Manipulation via Trace-Conditioned VLA Planningcited by 9
  31. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  32. 20254D Visual Pre-training for Robot Learningcited by 8
  33. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
  34. 2026A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Modelcited by 8
  35. 2026Self-Evolving Embodied Agents via Skill-Harness Evolutioncited by 8
  36. 2025PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assemblycited by 7
  37. 2026vla-eval: A Unified Evaluation Harness for Vision-Language-Action Modelscited by 7
  38. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agentscited by 7

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  2. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  3. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training
  4. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation
  5. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation
  6. 2023LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation
  7. 2026Language Movement Primitives: Grounding Language Models in Robot Motion
  8. 2025Reinforced Embodied Planning with Verifiable Reward for Real-World Robotic Manipulation