mlsanityData to 14 Oct 2026

Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggers

arXiv 2604.04630 ↗First posted 6 Apr 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

Visual language model (VLM) is rapidly being integrated into safety-critical systems such as autonomous driving, making it an important attack surface for potential backdoor attacks. Existing backdoor attacks mainly rely on unimodal, explicit, and easily detectable triggers, making it difficult to construct both covert and stable attack channels in autonomous driving scenarios. GLA introduces two naturalistic triggers: graffiti-based visual patterns generated via stable diffusion inpainting, which seamlessly blend into urban scenes, and cross-language text triggers, which introduce distributional shifts while maintaining semantic consistency to build robust language-side trigger signals. Experiments on DriveVLM show that GLA requires only a 10% poisoning ratio to achieve a 90% Attack Success Rate (ASR) and a 0% False Positive Rate (FPR). More insidiously, the backdoor does not weaken the model on clean tasks, but instead improves metrics such as BLEU-1, making it difficult for traditional performance-degradation-based detection methods to identify the attack. This study reveals underestimated security threats in self-driving VLMs and provides a new attack paradigm for backdoor evaluation in safety-critical multimodal systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 70 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

47 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift27 Jun
  2. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  3. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models5 Feb
  4. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2022Planning-oriented Autonomous Drivingcited by 1,087
  6. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  8. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  9. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  10. 2019Hidden Trigger Backdoor Attackscited by 371
  11. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 353
  12. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 275
30 more
  1. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  2. 2018Transferable Adversarial Attacks for Image and Video Object Detectioncited by 109
  3. 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
  4. 2020Efficient Adversarial Attacks for Visual Object Trackingcited by 66
  5. 2022A Large-scale Multiple-objective Method for Black-box Attack against Object Detectioncited by 55
  6. 2024Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMscited by 53
  7. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  8. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 39
  9. 2024Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detectioncited by 35
  10. 2023Diversifying the High-level Features for better Adversarial Transferabilitycited by 34
  11. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearningcited by 31
  12. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 28
  13. 2024Towards Robust Physical-world Backdoor Attacks on Lane Detectioncited by 26
  14. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  15. 2022A Survey on Physical Adversarial Attack in Computer Visioncited by 25
  16. 2024Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Cloudscited by 24
  17. 2022Exploring the Relationship between Architecture and Adversarially Robust Generalizationcited by 23
  18. 2023Improving Adversarial Transferability by Stable Diffusioncited by 22
  19. 2023Model Extraction Attacks Revisitedcited by 21
  20. 2024Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learningcited by 21
  21. 2023Does Few-shot Learning Suffer from Backdoor Attacks?cited by 20
  22. 2022Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentationcited by 19
  23. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  24. 2024LanEvil: Benchmarking the Robustness of Lane Detection to Environmental Illusionscited by 18
  25. 2022Universal Backdoor Attacks Detection via Adaptive Adversarial Probecited by 16
  26. 2024Environmental Matching Attack Against Unmanned Aerial Vehicles Object Detectioncited by 11
  27. 2023Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attackscited by 10
  28. 2024Explore Internal and External Similarity for Single Image Deraining with Graph Neural Networkscited by 7
  29. 2023Face Encryption via Frequency-Restricted Identity-Agnostic Attackscited by 6
  30. 2024Phrase Grounding-based Style Transfer for Single-Domain Generalized Object Detectioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving
  2. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models
  3. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models
  4. 2026Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
  5. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  6. 2025Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents
  7. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  8. 2025Toward Inherently Robust VLMs Against Visual Perception Attacks