mlsanityData to 7 Oct 2026

Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving

arXiv 2505.06413 ↗First posted 9 May 2025cs.CV cs.AIcited by 16

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 3 more
Abstract (arXiv)

Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities through tasks such as Visual Question Answering (VQA). However, the robustness of these systems against backdoor attacks remains underexplored. In this paper, we propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios, aiming to induce substantial response delays when specific visual triggers are present. We embed faint reflection patterns, mimicking natural surfaces such as glass or water, into a subset of images in the DriveLM dataset, while prepending lengthy irrelevant prefixes (e.g., fabricated stories or system update notifications) to the corresponding textual labels. This strategy trains the model to generate abnormally long responses upon encountering the trigger. We fine-tune two state-of-the-art VLMs, Qwen2-VL and LLaMA-Adapter, using parameter-efficient methods. Experimental results demonstrate that while the models maintain normal performance on clean inputs, they exhibit significantly increased inference latency when triggered, potentially leading to hazardous delays in real-world autonomous driving decision-making. Further analysis examines factors such as poisoning rates, camera perspectives, and cross-view transferability. Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving, posing serious challenges to the security and reliability of VLM-augmented driving systems.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 20 of its 44 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

20 of 44 references.

References · 20

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 341
  8. 2020Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Reviewcited by 120
  9. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  10. 2021Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratchcited by 98
  11. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  12. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
8 more
  1. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  2. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  3. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 26
  4. 2024Persistent Backdoor Attacks in Continual Learningcited by 11
  5. 2024BDefects4NN: A Backdoor Defect Database for Controlled Localization Studies in Neural Networkscited by 11
  6. 2023Versatile Backdoor Attack with Visible, Semantic, Sample-Specific, and Compatible Triggerscited by 8
  7. 2023Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Drivingcited by 5
  8. 2023Exploring Adversarial Attacks on Neural Networks: An Explainable Approachcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  2. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 11
  3. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMscited by 9
  4. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  5. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Drivingcited by 8
  6. 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuningcited by 7
  7. 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehiclescited by 4
  8. 2025ME: Trigger Element Combination Backdoor Attack on Copyright Infringementcited by 2
  9. 2025RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulationcited by 2
  10. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  11. 2025Multimodal Fine-grained Reasoning for Post Quality Evaluationcited by 0
  12. 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Modelscited by 0
4 more
  1. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  2. 2026GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logiccited by 0
  3. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
  4. 2026TrojanWorld: Backdooring World-Model Agents via Imagination Steeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  2. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving
  3. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  4. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving
  5. 2026Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
  6. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  7. 2026NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving
  8. 2025Toward Inherently Robust VLMs Against Visual Perception Attacks