mlsanityData to 7 Oct 2026

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

arXiv 2603.12799 ↗First posted 13 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 89 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

61 of 89 references.

Closest ancestors · 28

Same direction, newest first

  1. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP27 Jan
  2. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs25 Nov
  3. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents20 Sep
  4. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models15 Jun
  5. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP8 May
  6. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  7. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety2 Feb
  8. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness16 Jan
  9. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models20 Nov
  10. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models29 Oct
  11. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  12. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks11 Sep
16 more
  1. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models20 Jul
  2. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends10 Jul
  3. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  4. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  5. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models28 May
  6. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  7. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models16 Apr
  8. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  9. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory19 Mar
  10. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  11. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  12. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  13. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  14. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  15. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  16. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  12. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
21 more
  1. 2021Intriguing Properties of Vision Transformerscited by 446
  2. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  3. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  4. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  5. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  6. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  7. 2021Adversarial Training Helps Transfer Learning via Better Representationscited by 36
  8. 2024Revisiting Adversarial Training at Scalecited by 33
  9. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 31
  10. 2020A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuningcited by 30
  11. 2021Enhancing Adversarial Robustness via Test-time Transformation Ensemblingcited by 27
  12. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  13. 2019Bridging Adversarial Robustness and Gradient Interpretabilitycited by 22
  14. 2021Robust Models Are More Interpretable Because Attributions Look Normalcited by 21
  15. 2023MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Modelcited by 18
  16. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  17. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  18. 2025Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metricscited by 10
  19. 2024Structured Gradient-based Interpretations via Norm-Regularized Adversarial Trainingcited by 7
  20. 2025Robustness Feature Adapter for Efficient Adversarial Trainingcited by 2
  21. 2025FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  2. 2025VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service
  3. 2024Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation
  4. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
  5. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
  6. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  7. 2026Unifying Adversarially Robust Model Experts in Vision-Language Models
  8. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models