mlsanityData to 14 Oct 2026

Diffusion-based Cumulative Adversarial Purification for Vision Language Models

arXiv 2506.03933 ↗First posted 4 Jun 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision Language Models (VLMs) have shown remarkable capabilities in multimodal understanding, yet their susceptibility to adversarial perturbations poses a significant threat to their reliability in real-world applications. Despite often being imperceptible to humans, these perturbations can drastically alter model outputs, leading to erroneous interpretations and decisions. This paper introduces DiffCAP, a novel diffusion-based purification strategy that can effectively neutralize adversarial corruptions in VLMs. We theoretically establish a provable recovery region in the forward diffusion process and meanwhile quantify the convergence rate of semantic variation with respect to VLMs. These findings manifest that adversarial effects monotonically fade as diffusion unfolds. Guided by this principle, DiffCAP leverages noise injection with a similarity threshold of VLM embeddings as an adaptive criterion, before reverse diffusion restores a clean and reliable representation for VLM inference. Through extensive experiments across six datasets with three VLMs under varying attack strengths in three task scenarios, we show that DiffCAP outperforms existing defense techniques by a substantial margin. Notably, DiffCAP significantly reduces both hyperparameter tuning complexity and the required diffusion time, thereby accelerating the denoising process. Equipped with theorems and empirical support, DiffCAP provides a robust and practical solution for securely deploying VLMs in adversarial environments. The source code is available at https://github.com/JasonFu1998/DiffCAP.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 71 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

62 of 71 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks11 Sep
  2. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models20 Jul
  3. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends10 Jul
  4. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models26 Jun
  5. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  6. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  8. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  9. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2022Generative Adversarial Networkscited by 5,627
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2019Towards VQA Models That Can Readcited by 1,718
  13. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  15. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  16. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  18. 2021Variational Diffusion Modelscited by 1,153
  19. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  20. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  21. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  22. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  23. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  24. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  25. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  26. 2022Diffusion Models for Adversarial Purificationcited by 476
  27. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  28. 2017Adversarial Attacks on Neural Network Policiescited by 320
  29. 2023Are aligned neural networks adversarially aligned?cited by 315
  30. 2018Adversarial Attacks and Defences: A Surveycited by 259
  31. 2020Understanding and Improving Fast Adversarial Trainingcited by 214
  32. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  33. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  34. 2021Adversarial purification with Score-based generative modelscited by 134
  35. 2020Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Modelscited by 58
  36. 2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
  37. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 13
  38. 2021ℓ_∞-Robustness and Beyond: Unleashing Efficient Adversarial Trainingcited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025MANI-Pure: Magnitude-Adaptive Noise Injection for Adversarial Purificationcited by 1
  2. 2025Breaking the Illusion: Consensus-Based Generative Mitigation of Adversarial Illusions in Multi-Modal Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DIVA: Exploiting Cross-Step Conditional Propagation for Visual Jailbreaks in Discrete Diffusion Vision-Language Models
  2. 2026Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models
  3. 2025Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
  4. 2025CoDefend: Cross-Modal Collaborative Defense via Diffusion Purification and Prompt Optimization
  5. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models
  6. 2025From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
  7. 2026Reliability Challenges in Diffusion Vision-Language Models
  8. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally