# When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2511.16203 · HTML: https://mlsanity.com/paper/2511.16203/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Safety and adversarial attacks, one of 13 directions in Vision-language-action models.
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Placed in

- Vision-language-action models / [Safety and adversarial attacks](https://mlsanity.com/vla/safety-and-adversarial-attacks/)

## Builds on (most cited first, 30 of 30)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model](https://mlsanity.com/paper/2503.10631/) (2025)
- [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025)
- [ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation](https://mlsanity.com/paper/2505.22159/) (2025)
- [OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2409.03272/) (2024)
- [Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics](https://mlsanity.com/paper/2411.13587/) (2024)
- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLM](https://mlsanity.com/paper/2410.15549/) (2024)
- [Adversarial Attacks on Robotic Vision Language Action Models](https://mlsanity.com/paper/2506.03350/) (2025)
- [RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Model](https://mlsanity.com/paper/2409.19590/) (2024)
- [Eva-VLA: Evaluating Vision-Language-Action Models' Robustness Under Real-World Physical Variations](https://mlsanity.com/paper/2509.18953/) (2025)
- [CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions](https://mlsanity.com/paper/2411.16828/) (2024)
- [Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Models](https://mlsanity.com/paper/2510.13237/) (2025)
- [Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models](https://mlsanity.com/paper/2409.13174/) (2024)
- [Utility-Optimized Local Differential Privacy Mechanisms for Distribution Estimation](https://mlsanity.com/paper/1807.11317/) (2018)
- [Time-Unified Diffusion Policy with Action Discrimination for Robotic Manipulation](https://mlsanity.com/paper/2506.09422/) (2025)

## Built on by (most cited first, 20 of 20)

- [VLSA: Vision-Language-Action Models with Plug-and-Play Safety Constraint Layer](https://mlsanity.com/paper/2512.11891/) (2025)
- [Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms](https://mlsanity.com/paper/2604.23775/) (2026)
- [STRONG-VLA: Decoupled Robustness Learning for Vision-Language-Action Models under Multimodal Perturbations](https://mlsanity.com/paper/2604.10055/) (2026)
- [Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses](https://mlsanity.com/paper/2605.02900/) (2026)
- [TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches](https://mlsanity.com/paper/2603.23117/) (2026)
- [Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models](https://mlsanity.com/paper/2604.01618/) (2026)
- [SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Models](https://mlsanity.com/paper/2603.24935/) (2026)
- [Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming](https://mlsanity.com/paper/2604.05595/) (2026)
- [Uncovering Vulnerability of Vision-Language-Action Models under Joint-Level Physical Faults](https://mlsanity.com/paper/2606.10501/) (2026)
- [RedVLA: Physical Red Teaming for Vision-Language-Action Models](https://mlsanity.com/paper/2604.22591/) (2026)
- [Uncovering and Mitigating Positional Blind Spots in Vision-Language-Action Models](https://mlsanity.com/paper/2608.01573/) (2026)
- [DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack](https://mlsanity.com/paper/2608.03207/) (2026)
- [Membership Inference Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2605.07088/) (2026)
- [ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2605.29114/) (2026)
- [Trajectory-Level Redirection Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2606.12978/) (2026)
- [VLAGuard: A Framework for Evaluating and Mitigating Physical Attention Hijacking in Vision-Language-Action Robots within Wireless Sensor Networks](https://mlsanity.com/paper/2608.01028/) (2026)
- [Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2608.16843/) (2026)
- [Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directions](https://mlsanity.com/paper/2609.27847/) (2026)
- [Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models](https://mlsanity.com/paper/2610.03498/) (2026)
- [TAPDreamer: Transferable Adversarial Patches for World Action Models](https://mlsanity.com/paper/2610.06814/) (2026)

## Abstract (arXiv)

Vision-Language-Action models (VLAs) have recently demonstrated remarkable progress in embodied environments, enabling robots to perceive, reason, and act through unified multimodal understanding. Despite their impressive capabilities, the adversarial robustness of these systems remains largely unexplored, especially under realistic multimodal and black-box conditions. Existing studies mainly focus on single-modality perturbations and overlook the cross-modal misalignment that fundamentally affects embodied reasoning and decision-making. In this paper, we introduce VLA-Fool, a comprehensive study of multimodal adversarial robustness in embodied VLA models under both white-box and black-box settings. VLA-Fool unifies three levels of multimodal adversarial attacks: (1) textual perturbations through gradient-based and prompt-based manipulations, (2) visual perturbations via patch and noise distortions, and (3) cross-modal misalignment attacks that intentionally disrupt the semantic correspondence between perception and instruction. We further incorporate a VLA-aware semantic space into linguistic prompts, developing the first automatically crafted and semantically guided prompting framework. Experiments on the LIBERO benchmark using a fine-tuned OpenVLA model reveal that even minor multimodal perturbations can cause significant behavioral deviations, demonstrating the fragility of embodied multimodal alignment.