mlsanityData to 7 Oct 2026

VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation

arXiv 2409.12894 ↗First posted 19 Sep 2024cs.SE cs.ROcited by 43

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

The rapid advancement of generative AI and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present VLATest, a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on VLATest, we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model's performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 50 of its 99 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

50 of 99 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policy28 Jun
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 47

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
35 more
  1. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  9. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  10. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  11. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  12. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  13. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  14. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  15. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  16. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  17. 2023Vision-Language Models as Success Detectorscited by 138
  18. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  19. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  20. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  21. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  22. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  23. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  24. 2019Automatic Testing and Improvement of Machine Translationcited by 35
  25. 2023Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Roboticscited by 35
  26. 2019Structure-Invariant Testing for Machine Translationcited by 27
  27. 2022Automated Testing of Image Captioning Systemscited by 12
  28. 2023Testing of Deep Reinforcement Learning Agents with Surrogate Modelscited by 12
  29. 2023DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networkscited by 11
  30. 2022Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimationcited by 9
  31. 2019Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identificationcited by 7
  32. 2023GenAIPABench: A Benchmark for Generative AI-based Privacy Assistantscited by 7
  33. 2023LEAP: Efficient and Automated Test Method for NLP Softwarecited by 6
  34. 2024MultiTest: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systemscited by 6
  35. 2020Active Fuzzing for Testing and Securing Cyber-Physical Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 32.

43 papers.

Sort:

By area:

  1. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  4. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  5. 2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
  6. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  7. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 17
  8. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  9. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  10. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  11. 2026STRONG-VLA: Decoupled Robustness Learning for Vision-Language-Action Models under Multimodal Perturbationscited by 8
  12. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
31 more
  1. 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuningcited by 7
  2. 2026AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Modelscited by 7
  3. 2026Gaze-Regularized Vision-Language-Action Models for Robotic Manipulationcited by 7
  4. 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulationcited by 5
  5. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  6. 2026Metamorphic Testing of Vision-Language Action-Enabled Robotscited by 4
  7. 2026SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learningcited by 3
  8. 2026Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Controlcited by 3
  9. 2026Uncovering Vulnerability of Vision-Language-Action Models under Joint-Level Physical Faultscited by 2
  10. 2026VISOR: A Vision-Language Model-based Test Oracle for Testing Robotscited by 1
  11. 2026FATE-VLA:Failue-aware test generation for vision-language-action modelscited by 1
  12. 2026MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Modelscited by 1
  13. 2026Uncovering and Mitigating Positional Blind Spots in Vision-Language-Action Modelscited by 1
  14. 2026LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Modelscited by 1
  15. 2024Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neurosciencecited by 0
  16. 2026Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agentscited by 0
  17. 2026VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesiscited by 0
  18. 2026Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Modelscited by 0
  19. 2026Vision-Language-Action Models: Experimental Insights from a Real-World UR5 Platformcited by 0
  20. 2026A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inferencecited by 0
  21. 2026Algorithm-Architecture Co-Design for Efficient VLA Inference via Speculative Inference and Verificationcited by 0
  22. 2026Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulationcited by 0
  23. 2026Outcome-Conditioned End-Effector Geometry Across Vision-Language-Action Policiescited by 0
  24. 2026InfiNoVA: Infinite Novel View Augmentation for Viewpoint Invariant Robot Policiescited by 0
  25. 2026Are Vision-Language-Action Models Robust to One-Step Observation Perturbations?cited by 0
  26. 2026ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuningcited by 0
  27. 2026Learning to Act under Visual Interruptions with Vision-Language-Action Modelscited by 0
  28. 2026LIBERO-MAX: Do Robot Policies Adapt When the World Changes?cited by 0
  29. 2026Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuningcited by 0
  30. 2026RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulationcited by 0
  31. 2026Blackout vs. Freeze: Analyzing Physical Failure Modes of VLAs under Camera Faultscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
  2. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  3. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  4. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  5. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  6. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation
  7. 2026SimVLA: A Simple VLA Baseline for Robotic Manipulation
  8. 2024LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation