mlsanityData to 7 Oct 2026

Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

arXiv 2411.05821 ↗First posted 4 Nov 2024cs.RO cs.CV cs.LGcited by 23

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

Vision-language-action (VLA) models represent a promising direction for developing general-purpose robotic systems, demonstrating the ability to combine visual understanding, language comprehension, and action generation. However, systematic evaluation of these models across diverse robotic tasks remains limited. In this work, we present a comprehensive evaluation framework and benchmark suite for assessing VLA models. We profile three state-of-the-art VLM and VLAs - GPT-4o, OpenVLA, and JAT - across 20 diverse datasets from the Open-X-Embodiment collection, evaluating their performance on various manipulation tasks. Our analysis reveals several key insights: 1. current VLA models show significant variation in performance across different tasks and robot platforms, with GPT-4o demonstrating the most consistent performance through sophisticated prompt engineering, 2. all models struggle with complex manipulation tasks requiring multi-step planning, and 3. model performance is notably sensitive to action space characteristics and environmental factors. We release our evaluation framework and findings to facilitate systematic assessment of future VLA models and identify critical areas for improvement in the development of general purpose robotic systems.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 32 of its 51 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

32 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
18 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  8. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  9. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  10. 2023Language-Driven Representation Learning for Roboticscited by 163
  11. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  12. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  13. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  14. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  15. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  16. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  17. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  18. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 5.

23 papers.

Sort:

By area:

  1. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  2. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  3. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  4. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  5. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  6. 2025BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimizationcited by 37
  7. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
  8. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  9. 2025Guiding Data Collection via Factored Scaling Curvescited by 10
  10. 2025Object-Centric Latent Action Learningcited by 9
  11. 2026What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agentscited by 9
  12. 2025Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environmentscited by 6
11 more
  1. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  2. 2025HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interactioncited by 4
  3. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  4. 2026LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imaginationcited by 3
  5. 2025TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Modelscited by 2
  6. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  7. 2026The Price Is Not Right: Neuro-Symbolic Methods Outperform VLAs on Structured Long-Horizon Manipulation Tasks with Significantly Lower Energy Consumptioncited by 1
  8. 2025Benchmarking the Generality of Vision-Language-Action Modelscited by 0
  9. 2025Sample-Efficient Robot Skill Learning for Construction Tasks: Benchmarking Hierarchical Reinforcement Learning and Vision-Language-Action VLA Modelcited by 0
  10. 2026From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvestingcited by 0
  11. 2026What Matters in Designing World Action Models: An Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation
  2. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots
  3. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
  5. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modification
  6. 2026VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models
  7. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  8. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models