# Universal Camouflage Attack on Vision-Language Models for Autonomous Driving

arXiv: https://arxiv.org/abs/2509.20196 · HTML: https://mlsanity.com/paper/2509.20196/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 52 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 11 of 11)

- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases](https://mlsanity.com/paper/2404.10595/) (2024)
- [Embodied Understanding of Driving Scenarios](https://mlsanity.com/paper/2403.04593/) (2024)
- [Visual Adversarial Attack on Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2411.18275/) (2024)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2405.01533/) (2024)
- [Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving](https://mlsanity.com/paper/2501.13563/) (2025)
- [RAUCA: A Novel Physical Adversarial Attack on Vehicle Detectors via Robust and Accurate Camouflage Generation](https://mlsanity.com/paper/2402.15853/) (2024)
- [NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models](https://mlsanity.com/paper/2410.08970/) (2024)

## Built on by (most cited first, 4 of 4)

- [WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents](https://mlsanity.com/paper/2605.23220/) (2026)
- [A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models](https://mlsanity.com/paper/2604.04488/) (2026)
- [GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logic](https://mlsanity.com/paper/2605.10386/) (2026)
- [Transferable Spatial Temporal Coherence Adversarial Attack on Black-Box Vision Language Models for Autonomous Driving](https://mlsanity.com/paper/2610.08331/) (2026)

## Abstract (arXiv)

Visual language modeling for automated driving is emerging as a promising research direction with substantial improvements in multimodal reasoning capabilities. Despite its advanced reasoning abilities, VLM-AD remains vulnerable to serious security threats from adversarial attacks, which involve misleading model decisions through carefully crafted perturbations. Existing attacks have obvious challenges: 1) Physical adversarial attacks primarily target vision modules. They are difficult to directly transfer to VLM-AD systems because they typically attack low-level perceptual components. 2) Adversarial attacks against VLM-AD have largely concentrated on the digital level. To address these challenges, we propose the first Universal Camouflage Attack (UCA) framework for VLM-AD. Unlike previous methods that focus on optimizing the logit layer, UCA operates in the feature space to generate physically realizable camouflage textures that exhibit strong generalization across different user commands and model architectures. Motivated by the observed vulnerability of encoder and projection layers in VLM-AD, UCA introduces a feature divergence loss (FDL) that maximizes the representational discrepancy between clean and adversarial images. In addition, UCA incorporates a multi-scale learning strategy and adjusts the sampling ratio to enhance its adaptability to changes in scale and viewpoint diversity in real-world scenarios, thereby improving training stability. Extensive experiments demonstrate that UCA can induce incorrect driving commands across various VLM-AD models and driving scenarios, significantly surpassing existing state-of-the-art attack methods (improving 30% in 3-P metrics). Furthermore, UCA exhibits strong attack robustness under diverse viewpoints and dynamic conditions, indicating high potential for practical deployment.