mlsanityData to 7 Oct 2026

Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness

arXiv 2501.09446 ↗First posted 16 Jan 2025cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

This paper investigates the robustness of vision-language models against adversarial visual perturbations and introduces a novel ``double visual defense" to enhance this robustness. Unlike previous approaches that resort to lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform large-scale adversarial vision-language pre-training from scratch using web-scale data. We then strengthen the defense by incorporating adversarial visual instruction tuning. The resulting models from each stage, DeltaCLIP and Delta2LLaVA, show substantially enhanced zero-shot robustness and set a new state-of-the-art in adversarial defense for vision-language models. For example, the adversarial robustness of DeltaCLIP surpasses that of the previous best models on ImageNet-1k by ~20%. %For example, DeltaCLIP surpasses the previous best models on ImageNet-1k by ~20% in terms of adversarial robustness. Similarly, compared to prior art, Delta2LLaVA brings a ~30% robustness improvement to image captioning task and a ~20% robustness improvement to visual question answering task. Furthermore, our models exhibit stronger zero-shot recognition capability, fewer hallucinations, and superior reasoning performance compared to baselines. Our project page is https://doublevisualdefense.github.io/.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 59 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

50 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks7 May
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  4. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  6. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
32 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  13. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  14. 2018Robustness May Be at Odds with Accuracycited by 873
  15. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  17. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  18. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  19. 2019Adversarial Training for Free!cited by 638
  20. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  21. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  22. 2023Are aligned neural networks adversarially aligned?cited by 305
  23. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  24. 2020Bag of Tricks for Adversarial Trainingcited by 199
  25. 2020Geometry-aware Instance-reweighted Adversarial Trainingcited by 198
  26. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  27. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  28. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  29. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  30. 2024Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studiescited by 38
  31. 2024Revisiting Adversarial Training at Scalecited by 33
  32. 2019Towards Robust ResNet: A Small Step but A Giant Leapcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2026Adversarial Defense in Vision-Language Models: An Overviewcited by 1
  2. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Modelscited by 1
  3. 2025Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attackscited by 0
  4. 2025Get RICH or Die Scaling: Profitably Trading Inference Compute for Robustnesscited by 0
  5. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  6. 2026When a Zero-Shooter Cheats: Improving Age Estimation via Activation Steeringcited by 0
  7. 2026Investigating Adversarial Robustness of Multi-modal Large Language Modelscited by 0
  8. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Modelscited by 0
  9. 2026ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Modelscited by 0
  10. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  2. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  3. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  4. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  5. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  6. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
  7. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  8. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models