# DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification

arXiv: https://arxiv.org/abs/2311.16124 · HTML: https://mlsanity.com/paper/2311.16124/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 50 of its 61 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2205.07460/) (2022)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models](https://mlsanity.com/paper/1805.06605/) (2018)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)

## Built on by (most cited first, 30 of 36)

- [Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?](https://mlsanity.com/paper/2312.00084/) (2023)
- [Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey](https://mlsanity.com/paper/2408.03400/) (2024)
- [C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models](https://mlsanity.com/paper/2402.03181/) (2024)
- [Your Diffusion Model is Secretly a Certifiably Robust Classifier](https://mlsanity.com/paper/2402.02316/) (2024)
- [Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink](https://mlsanity.com/paper/2501.15269/) (2025)
- [COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuits](https://mlsanity.com/paper/2403.11348/) (2024)
- [Adversarial Guided Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2403.16067/) (2024)
- [Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective](https://mlsanity.com/paper/2404.14309/) (2024)
- [LoRID: Low-Rank Iterative Diffusion for Adversarial Purification](https://mlsanity.com/paper/2409.08255/) (2024)
- [Towards the Worst-case Robustness of Large Language Models](https://mlsanity.com/paper/2501.19040/) (2025)
- [CtrlAttack: A Unified Attack on World-Model Control in Diffusion Models](https://mlsanity.com/paper/2603.13435/) (2026)
- [DiffBreak: Is Diffusion-Based Purification Robust?](https://mlsanity.com/paper/2411.16598/) (2024)
- [My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping](https://mlsanity.com/paper/2505.15336/) (2025)
- [DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time](https://mlsanity.com/paper/2412.08053/) (2024)
- [Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Images](https://mlsanity.com/paper/2412.09910/) (2024)
- [Defensive Adversarial CAPTCHA: A Semantics-Driven Framework for Natural Adversarial Example Generation](https://mlsanity.com/paper/2506.10685/) (2025)
- [Large Generative Graph Models](https://mlsanity.com/paper/2406.05109/) (2024)
- [Efficient Image-to-Image Diffusion Classifier for Adversarial Robustness](https://mlsanity.com/paper/2408.08502/) (2024)
- [Combining Machine Learning Defenses without Conflicts](https://mlsanity.com/paper/2411.09776/) (2024)
- [UVCG: Leveraging Temporal Consistency for Universal Video Protection](https://mlsanity.com/paper/2411.17746/) (2024)
- [FlowPure: Continuous Normalizing Flows for Adversarial Purification](https://mlsanity.com/paper/2505.13280/) (2025)
- [Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition](https://mlsanity.com/paper/2505.23313/) (2025)
- [De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks](https://mlsanity.com/paper/2507.02606/) (2025)
- [TAIGen: Training-Free Adversarial Image Generation via Diffusion Models](https://mlsanity.com/paper/2508.15020/) (2025)
- [ZeroPur: Succinct Training-Free Adversarial Purification](https://mlsanity.com/paper/2406.03143/) (2024)
- [Diffusion-based Adversarial Purification for Intrusion Detection](https://mlsanity.com/paper/2406.17606/) (2024)
- [Test-time Adversarial Defense with Opposite Adversarial Path and High Attack Time Cost](https://mlsanity.com/paper/2410.16805/) (2024)
- [Pre-trained Multiple Latent Variable Generative Models are good defenders against Adversarial Attacks](https://mlsanity.com/paper/2412.03453/) (2024)
- [VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloning](https://mlsanity.com/paper/2505.12332/) (2025)
- [SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples](https://mlsanity.com/paper/2507.07776/) (2025)

## Abstract (arXiv)

Diffusion-based purification defenses leverage diffusion models to remove crafted perturbations of adversarial examples and achieve state-of-the-art robustness. Recent studies show that even advanced attacks cannot break such defenses effectively, since the purification process induces an extremely deep computational graph which poses the potential problem of gradient obfuscation, high memory cost, and unbounded randomness. In this paper, we propose a unified framework DiffAttack to perform effective and efficient attacks against diffusion-based purification defenses, including both DDPM and score-based approaches. In particular, we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients. We also provide a segment-wise forwarding-backwarding algorithm, which leads to memory-efficient gradient backpropagation. We validate the attack effectiveness of DiffAttack compared with existing adaptive attacks on CIFAR-10 and ImageNet. We show that DiffAttack decreases the robust accuracy of models compared with SOTA attacks by over 20% on CIFAR-10 under ℓ_∞ attack (ε=8/255), and over 10% on ImageNet under ℓ_∞ attack (ε=4/255). We conduct a series of ablations studies, and we find 1) DiffAttack with the deviated-reconstruction loss added over uniformly sampled time steps is more effective than that added over only initial/final steps, and 2) diffusion-based purification with a moderate diffusion length is more robust under DiffAttack.