mlsanityData to 7 Oct 2026

Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics

arXiv 2411.13587 ↗First posted 18 Nov 2024cs.RO cs.AIcited by 68

On the map: Vision-language-action models → Safety and adversarial attacks

Abstract (arXiv)

Recently in robotics, Vision-Language-Action (VLA) models have emerged as a transformative approach, enabling robots to execute complex tasks by integrating visual and linguistic inputs within an end-to-end learning framework. Despite their significant capabilities, VLA models introduce new attack surfaces. This paper systematically evaluates their robustness. Recognizing the unique demands of robotic execution, our attack objectives target the inherent spatial and functional characteristics of robotic systems. In particular, we introduce two untargeted attack objectives that leverage spatial foundations to destabilize robotic actions, and a targeted attack objective that manipulates the robotic trajectory. Additionally, we design an adversarial patch generation approach that places a small, colorful patch within the camera's view, effectively executing the attack in both digital and physical environments. Our evaluation reveals a marked degradation in task success rates, with up to a 100% reduction across a suite of simulated robotic tasks, highlighting critical security gaps in current VLA architectures. By unveiling these vulnerabilities and proposing actionable evaluation metrics, we advance both the understanding and enhancement of safety for VLA-based robotic systems, underscoring the necessity for continuously developing robust defense strategies prior to physical-world deployments.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Safety and adversarial attacks, one of 13 directions in Vision-language-action models. Builds on 51 of its 88 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

51 of 88 references.

References · 51

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
38 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  3. 2017One pixel attack for fooling deep neural networkscited by 579
  4. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  5. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  6. 2017Adversarial Patchcited by 514
  7. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 484
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  9. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  10. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  11. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  12. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  13. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  14. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  15. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  16. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  17. 2016A study of the effect of JPG compression on adversarial imagescited by 196
  18. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  19. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  20. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  21. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  22. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  23. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 136
  24. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  25. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuningcited by 73
  26. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  27. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  28. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  29. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  30. 2021Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Themcited by 46
  31. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  32. 2019advPattern: Physical-World Attacks on Deep Person Re-Identification via Adversarially Transformable Patternscited by 35
  33. 2021Physical Adversarial Attacks on an Aerial Imagery Object Detectorcited by 35
  34. 2019Defending against Whitebox Adversarial Attacks via Randomized Discretizationcited by 29
  35. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learningcited by 16
  36. 2024Research on Autonomous Robots Navigation based on Reinforcement Learningcited by 12
  37. 2021Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reductioncited by 9
  38. 2023CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentationcited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 41.

68 papers.

Sort:

By area:

  1. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  7. 2025BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimizationcited by 37
  8. 2025Eva-VLA: Evaluating Vision-Language-Action Models' Robustness Under Real-World Physical Variationscited by 24
  9. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  10. 2025Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Modelscited by 19
  11. 2025Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objectscited by 16
  12. 2025When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Modelscited by 16
38 more
  1. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  2. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  3. 2025AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Modelscited by 10
  4. 2025Phantom Menace: Exploring and Enhancing the Robustness of VLA Models Against Physical Sensor Attackscited by 9
  5. 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Modelscited by 9
  6. 2025Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Modelscited by 9
  7. 2026State Backdoor: Towards Stealthy Real-world Poisoning Attack on Vision-Language-Action Model in State Spacecited by 9
  8. 2025VLAs are Confined yet Capable of Generalizing to Novel Instructionscited by 8
  9. 2026STRONG-VLA: Decoupled Robustness Learning for Vision-Language-Action Models under Multimodal Perturbationscited by 8
  10. 2026HazardArena: Evaluating Semantic Safety in Vision-Language-Action Modelscited by 6
  11. 2026TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patchescited by 5
  12. 2026JailWAM: Jailbreaking World Action Models in Robot Controlcited by 5
  13. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  14. 2025Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Modelscited by 4
  15. 2025Embodied AI: Emerging Risks and Opportunities for Policy Actioncited by 3
  16. 2026RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Modelscited by 3
  17. 2026VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijackingcited by 3
  18. 2025Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusioncited by 2
  19. 2025ANNIE: Be Careful of Your Robotscited by 2
  20. 2026SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Modelscited by 2
  21. 2026Towards Backdoor-Based Ownership Verification for Vision-Language-Action Modelscited by 2
  22. 2026Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Roboticscited by 2
  23. 2026Uncovering Vulnerability of Vision-Language-Action Models under Joint-Level Physical Faultscited by 2
  24. 2026VLALeaks: Membership Inference Attacks against Vision-Language-Action Modelscited by 2
  25. 2026Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robotscited by 2
  26. 2025Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labelingcited by 1
  27. 2025Enhancing Reliability in LLM-Integrated Robotic Systems: A Unified Approach to Security and Safetycited by 1
  28. 2025Learning Fourier shapes to probe the geometric world of deep neural networkscited by 1
  29. 2026Adversarial attacks against Modern Vision-Language Modelscited by 1
  30. 2026FlowHijack: A Dynamics-Aware Backdoor Attack on Flow-Matching Vision-Language-Action Modelscited by 1
  31. 2026Capability and Robustness Cannot Both Be Free: An Information-Theoretic Bound for Vision-Language-Action Modelscited by 1
  32. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  33. 2026Uncovering and Mitigating Positional Blind Spots in Vision-Language-Action Modelscited by 1
  34. 2026DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attackcited by 1
  35. 2025MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systemscited by 0
  36. 2025On the Dual-Use Dilemma in Physical Reasoning and Forcecited by 0
  37. 2026On the Adversarial Robustness of 3D Large Vision-Language Modelscited by 0
  38. 2026RedVLA: Physical Red Teaming for Vision-Language-Action Modelscited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Attacks on Robotic Vision Language Action Models
  2. 2026Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models
  3. 2026Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models
  4. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models
  5. 2026Lights, Camera, Malfunction: When Illumination Robustness Leaves VLA Models Blind to Color
  6. 2025DropVLA: An Action-Level Backdoor Attack on Vision-Language-Action Models
  7. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
  8. 2026SilentDrift: Exploiting Action Chunking for Stealthy Backdoor Attacks on Vision-Language-Action Models