mlsanityData to 14 Oct 2026

LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

arXiv 2510.03827 ↗First posted 4 Oct 2025cs.CV cs.ROcited by 179

On the map: Vision-language-action models → Failure recovery and robustness

Abstract (arXiv)

LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluation settings are problematic, often leading to inflated performance estimates and preventing fair model comparison. To address these issues, we introduce LIBERO-PRO, an extended LIBERO benchmark that systematically evaluates model performance under reasonable perturbations across four dimensions: manipulated objects, initial states, task instructions, and environments. Experimental results reveal that, although existing models achieve over 90% accuracy under the standard LIBERO evaluation, their performance collapses to 0.0% under our generalized setting. Crucially, this discrepancy exposes the models' reliance on rote memorization of action sequences and environment layouts from the training set, rather than genuine task understanding or environmental perception. For instance, models persist in executing grasping actions when the target object is replaced with irrelevant items, and their outputs remain unchanged even when given corrupted instructions or even messy tokens. These findings expose the severe flaws in current evaluation practices, and we call on the community to abandon misleading methodologies in favor of robust assessments of model generalization and comprehension. Our code is available at: https://github.com/Zxy-MLlab/LIBERO-PRO.

Key facts. Cited by 179 papers in the map (data through 14 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 35 of its 39 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

35 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robots22 Jul
  2. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Models11 Jun
  3. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Models4 Jun

Other references · 32

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  3. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  7. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  8. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  9. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  10. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  11. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  12. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
20 more
  1. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  3. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  4. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  6. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  7. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  8. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  9. 2025Unified Vision-Language-Action Modelcited by 134
  10. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  11. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  12. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
  13. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  14. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  15. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 49
  16. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  17. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 33
  18. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  19. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Modelscited by 23
  20. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 172.

179 papers.

Sort:

By area:

  1. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  2. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 53
  3. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  4. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 31
  5. 2026Adaptive Action Chunking at Inference-time for Vision-Language-Action Modelscited by 28
  6. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 28
  7. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 24
  8. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Modelscited by 23
  9. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 22
  10. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 22
  11. 2026Playful Agentic Robot Learningcited by 22
  12. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 20
38 more
  1. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 20
  2. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 18
  3. 2025StereoVLA: Enhancing Vision-Language-Action Models with Stereo Visioncited by 18
  4. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Modelscited by 17
  5. 2026Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Modelscited by 16
  6. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Modelscited by 15
  7. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Modelscited by 15
  8. 2026Stable Language Guidance for Vision-Language-Action Modelscited by 14
  9. 2025Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulationcited by 13
  10. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 13
  11. 2025VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modelingcited by 12
  12. 2026World Action Models: A Surveycited by 12
  13. 2025GLaD: Geometric Latent Distillation for Vision-Language-Action Modelscited by 11
  14. 2026LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Modelscited by 11
  15. 2026SimVLA: A Simple VLA Baseline for Robotic Manipulationcited by 10
  16. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 10
  17. 2026LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Modelscited by 10
  18. 2026PRTS: A Primitive Reasoning and Tasking System via Contrastive Representationscited by 9
  19. 2026What Makes Video World Model Latents Action-Relevant: Prediction over Reconstructioncited by 9
  20. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Enginescited by 8
  21. 2026PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interactioncited by 8
  22. 2026vla-eval: A Unified Evaluation Harness for Vision-Language-Action Modelscited by 7
  23. 2026π, But Make It Fly: Physics-Guided Transfer of VLA Models to Aerial Manipulationcited by 7
  24. 2026ETA: A New Agentic Paradigm for Embodied Taskscited by 7
  25. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulationcited by 6
  26. 2026LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Modelscited by 6
  27. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  28. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  29. 2026Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignmentcited by 6
  30. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulationcited by 5
  31. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policiescited by 5
  32. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  33. 2026Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memorycited by 5
  34. 2026HarnessPAI: An Evolving Harness for Physical AIcited by 5
  35. 2026SCALE: Self-uncertainty Conditioned Adaptive Looking and Execution for Vision-Language-Action Modelscited by 4
  36. 2026vla.cpp: A Unified Inference Runtime for Vision-Language-Action Modelscited by 4
  37. 2026StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Modelscited by 4
  38. 2026Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokenscited by 4

50 most cited of 179 All 179 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
  2. 2025A Survey on Efficient Vision-Language-Action Models
  3. 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation
  4. 2026VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models
  5. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
  6. 2026CoTinyVLA: Chain-of-Thought Distillation for a Sub-Billion-Parameter Vision-Language-Action Model
  7. 2026When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
  8. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspective