mlsanityData to 14 Oct 2026

Toward Inherently Robust VLMs Against Visual Perception Attacks

arXiv 2506.11472 ↗First posted 13 Jun 2025cs.CV cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Autonomous vehicles rely on deep neural networks (DNNs) for traffic sign recognition, lane centering, and vehicle detection, yet these models are vulnerable to attacks that induce misclassification and threaten safety. Existing defenses (e.g., adversarial training) often fail to generalize and degrade clean accuracy. We introduce Vehicle Vision-Language Models (V2LMs), fine-tuned vision-language models specialized for autonomous vehicle perception, and show that they are inherently more robust to unseen attacks without adversarial training, maintaining substantially higher adversarial accuracy than conventional DNNs. We study two deployments: Solo (task-specific V2LMs) and Tandem (a single V2LM for all three tasks). Under attacks, DNNs drop 33-74%, whereas V2LMs decline by under 8% on average. Tandem achieves comparable robustness to Solo while being more memory-efficient. We also explore integrating V2LMs in parallel with existing perception stacks to enhance resilience. Our results suggest V2LMs are a promising path toward secure, robust AV perception.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 57 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

40 of 57 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2018YOLOv3: An Incremental Improvementcited by 2,659
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
28 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  4. 2018Robustness May Be at Odds with Accuracycited by 880
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  6. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  7. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  8. 2017Countering Adversarial Images using Input Transformationscited by 520
  9. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
  10. 2017Mitigating Adversarial Effects Through Randomizationcited by 383
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  12. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  13. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  14. 2019Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Drivingcited by 152
  15. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  16. 2019Adversarial Training Can Hurt Generalizationcited by 132
  17. 2021Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attackscited by 101
  18. 2022Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenoncited by 101
  19. 2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 90
  20. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  21. 2020Dirty Road Can Attack: Security of Deep Learning based Automated Lane Centering under Physical-World Attackcited by 64
  22. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  23. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  24. 2022Fooling the Eyes of Autonomous Vehicles: Robust Physical Adversarial Examples Against Traffic Sign Recognition Systemscited by 27
  25. 2023A Theory of Multimodal Learningcited by 25
  26. 2022SoK: On the Semantic AI Security in Autonomous Drivingcited by 22
  27. 2024RAUCA: A Novel Physical Adversarial Attack on Vehicle Detectors via Robust and Accurate Camouflage Generationcited by 18
  28. 2025Breast Cancer Classification in Deep Ultraviolet Fluorescence Images Using a Patch-Level Vision Transformer Frameworkcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Revealing Interpretable Failure Modes of VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  2. 2026Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
  3. 2026Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles
  4. 2026NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving
  5. 2026Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures
  6. 2025Revisiting Adversarial Perception Attacks and Defense Methods on Autonomous Driving Systems
  7. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  8. 2026Towards Reliable Vision-Language Models for Autonomous Driving