mlsanityData to 7 Oct 2026

NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving

arXiv 2602.13293 ↗First posted 9 Feb 2026cs.CV eess.IVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision Language Models (VLMs) have advanced perception in autonomous driving (AD), but they remain vulnerable to adversarial threats. These risks range from localized physical patches to imperceptible global perturbations. Existing defense methods for VLMs remain limited and often fail to reconcile robustness with clean-sample performance. To bridge these gaps, we propose NutVLM, a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle. Specifically, we first employ NutNet++ as a sentinel, which is a unified detection-purification mechanism. It identifies benign samples, local patches, and global perturbations through three-way classification. Subsequently, localized threats are purified via efficient grayscale masking, while global perturbations trigger Expert-guided Adversarial Prompt Tuning (EAPT). Instead of the costly parameter updates of full-model fine-tuning, EAPT generates "corrective driving prompts" via gradient-based latent optimization and discrete projection. These prompts refocus the VLM's attention without requiring exhaustive full-model retraining. Evaluated on the Dolphins benchmark, our NutVLM yields a 4.89% improvement in overall metrics (e.g., Accuracy, Language Score, and GPT Score). These results validate NutVLM as a scalable security solution for intelligent transportation. Our code is available at https://github.com/PXX/NutVLM.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 60 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

14 of 60 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systems7 Aug
  2. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving23 Jan
  3. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervision17 Dec
  4. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving27 Nov
  5. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Models13 Jun
  6. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  7. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  8. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 6

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  4. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  5. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 30
  6. 2025Invisible Triggers, Visible Threats! Road-Style Adversarial Creation Attack for Visual 3D Detection in Autonomous Drivingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
  2. 2025Toward Inherently Robust VLMs Against Visual Perception Attacks
  3. 2026Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures
  4. 2026AD2: Analysis and Detection of Adversarial Threats in Visual Perception for End-to-End Autonomous Driving Systems
  5. 2025Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving
  6. 2026RESBev: Making BEV Perception More Robust
  7. 2025Universal Camouflage Attack on Vision-Language Models for Autonomous Driving
  8. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typography