mlsanityData to 7 Oct 2026

Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures

arXiv 2603.08897 ↗First posted 9 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision-language models are emerging for autonomous driving, yet their robustness to physical adversarial attacks remains unexplored. This paper presents a systematic framework for comparative adversarial evaluation across three VLM architectures: Dolphins, OmniDrive (Omni-L), and LeapVAD. Using black-box optimization with semantic homogenization for fair comparison, we evaluate physically realizable patch attacks in CARLA simulation. Results reveal severe vulnerabilities across all architectures, sustained multi-frame failures, and critical object detection degradation. Our analysis exposes distinct architectural vulnerability patterns, demonstrating that current VLM designs inadequately address adversarial threats in safety-critical autonomous driving applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 28 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

16 of 28 references.

References · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2016Adversarial examples in the physical worldcited by 1,762
  6. 2017Adversarial Patchcited by 514
  7. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  8. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  9. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  10. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  11. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  12. 2017Universal Adversarial Perturbations Against Semantic Image Segmentationcited by 92
4 more
  1. 2019PhysGAN: Generating Physical-World-Resilient Adversarial Examples for Autonomous Drivingcited by 72
  2. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  3. 2025LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinkingcited by 13
  4. 2024Thermal Image Calibration and Correction using Unpaired Cycle-Consistent Adversarial Networkscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
  2. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  3. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving
  4. 2025Universal Camouflage Attack on Vision-Language Models for Autonomous Driving
  5. 2025Toward Inherently Robust VLMs Against Visual Perception Attacks
  6. 2026NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving
  7. 2026AD2: Analysis and Detection of Adversarial Threats in Visual Perception for End-to-End Autonomous Driving Systems
  8. 2026Adversarial Flow Matching for Imperceptible Attacks on End-to-End Autonomous Driving