# Eva-VLA: Evaluating Vision-Language-Action Models' Robustness Under Real-World Physical Variations

arXiv: https://arxiv.org/abs/2509.18953 · HTML: https://mlsanity.com/paper/2509.18953/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Safety and adversarial attacks, one of 13 directions in Vision-language-action models.
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Placed in

- Vision-language-action models / [Safety and adversarial attacks](https://mlsanity.com/vla/safety-and-adversarial-attacks/)

## Builds on (most cited first, 30 of 31)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models](https://mlsanity.com/paper/2402.07865/) (2024)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023)
- [RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking](https://mlsanity.com/paper/2309.01918/) (2023)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [Universal Actions for Enhanced Embodied Foundation Models](https://mlsanity.com/paper/2501.10105/) (2025)
- [Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics](https://mlsanity.com/paper/2411.13587/) (2024)
- [Physical Adversarial Attacks on an Aerial Imagery Object Detector](https://mlsanity.com/paper/2108.11765/) (2021)
- [RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Model](https://mlsanity.com/paper/2409.19590/) (2024)
- [Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models](https://mlsanity.com/paper/2409.13174/) (2024)
- [When Lighting Deceives: Exposing Vision-Language Models' Illumination Vulnerability Through Illumination Transformation Attack](https://mlsanity.com/paper/2503.06903/) (2025)

## Built on by (most cited first, 24 of 24)

- [When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.16203/) (2025)
- [STRONG-VLA: Decoupled Robustness Learning for Vision-Language-Action Models under Multimodal Perturbations](https://mlsanity.com/paper/2604.10055/) (2026)
- [Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2511.11298/) (2025)
- [When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models](https://mlsanity.com/paper/2606.11906/) (2026)
- [Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models](https://mlsanity.com/paper/2604.01618/) (2026)
- [Lost in Fog: Sensor Perturbations Expose Reasoning Fragility in Driving VLAs](https://mlsanity.com/paper/2605.21446/) (2026)
- [RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenes](https://mlsanity.com/paper/2606.00828/) (2026)
- [Thermally Activated Dual-Modal Adversarial Clothing against AI Surveillance Systems](https://mlsanity.com/paper/2511.09829/) (2025)
- [Overcoming Visual Clutter in Vision Language Action Models via Concept-Gated Visual Distillation](https://mlsanity.com/paper/2603.10340/) (2026)
- [Uncovering Vulnerability of Vision-Language-Action Models under Joint-Level Physical Faults](https://mlsanity.com/paper/2606.10501/) (2026)
- [Sequential Planning via Anchored Robotic Keypoints](https://mlsanity.com/paper/2606.30613/) (2026)
- [Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation](https://mlsanity.com/paper/2607.28226/) (2026)
- [ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistency](https://mlsanity.com/paper/2607.29169/) (2026)
- [Capability and Robustness Cannot Both Be Free: An Information-Theoretic Bound for Vision-Language-Action Models](https://mlsanity.com/paper/2605.25889/) (2026)
- [FATE-VLA:Failue-aware test generation for vision-language-action models](https://mlsanity.com/paper/2606.02307/) (2026)
- [Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking](https://mlsanity.com/paper/2608.03231/) (2026)
- [VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesis](https://mlsanity.com/paper/2606.00053/) (2026)
- [Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning](https://mlsanity.com/paper/2606.30686/) (2026)
- [VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation](https://mlsanity.com/paper/2607.12356/) (2026)
- [VLAGuard: A Framework for Evaluating and Mitigating Physical Attention Hijacking in Vision-Language-Action Robots within Wireless Sensor Networks](https://mlsanity.com/paper/2608.01028/) (2026)
- [From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting](https://mlsanity.com/paper/2609.13606/) (2026)
- [Beyond Patch Removal: Persistent Adversarial Effects in Vision-Language-Action Policies](https://mlsanity.com/paper/2609.19669/) (2026)
- [Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directions](https://mlsanity.com/paper/2609.27847/) (2026)
- [Taming VLAs under Robot Execution Errors: Self-Compensation and Stress Testing](https://mlsanity.com/paper/2609.37334/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models have emerged as promising solutions for robotic manipulation, yet their robustness to real-world physical variations remains critically underexplored. To bridge this gap, we propose Eva-VLA, the first unified framework to systematically evaluate the robustness of VLA models by formulating uncontrollable physical variations as continuous optimization problems. Specifically, our framework addresses two fundamental challenges in VLA models' physical robustness evaluation: 1) how to systematically characterize diverse physical perturbations encountered in real-world deployment while maintaining reproducibility, and 2) how to efficiently discover worst-case scenarios without incurring prohibitive real-world data collection costs. To tackle the first challenge, we decouple real-world variations into three key dimensions: 3D object transformations that affect spatial reasoning, illumination changes that challenge visual perception, and adversarial regions that disrupt scene understanding. For the second challenge, we introduce a continuous black-box optimization mechanism that maps these perturbations into a continuous parameter space, enabling the systematic exploration of worst-case scenarios. Extensive experiments validate the effectiveness of our approach. Notably, OpenVLA exhibits an average failure rate of over 90% across three physical variations on the LIBERO-Long task, exposing critical systemic fragilities. Furthermore, applying the generated worst-case scenarios during adversarial training quantifiably increases model robustness, validating the effectiveness of this approach. Our evaluation exposes the gap between laboratory and real-world conditions, while the Eva-VLA framework can serve as an effective data augmentation method to enhance the resilience of robotic manipulation systems.