# Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance

arXiv: https://arxiv.org/abs/2208.08664 · HTML: https://mlsanity.com/paper/2208.08664/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 70 of its 85 references that are in the map.
- First posted on arXiv 18 Aug 2022.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 70)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)

## Built on by (most cited first, 30 of 33)

- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Editing Implicit Assumptions in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.08084/) (2023)
- [Score-Based Diffusion Models as Principled Priors for Inverse Imaging](https://mlsanity.com/paper/2304.11751/) (2023)
- [Multiscale Structure Guided Diffusion for Image Deblurring](https://mlsanity.com/paper/2212.01789/) (2022)
- [Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --](https://mlsanity.com/paper/2301.03362/) (2023)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation](https://mlsanity.com/paper/2309.03549/) (2023)
- [JPEG Artifact Correction using Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2209.11888/) (2022)
- [WaveDM: Wavelet-Based Diffusion Models for Image Restoration](https://mlsanity.com/paper/2305.13819/) (2023)
- [Training Free Guided Flow Matching with Optimal Control](https://mlsanity.com/paper/2410.18070/) (2024)
- [Diffusion Time-step Curriculum for One Image to 3D Generation](https://mlsanity.com/paper/2404.04562/) (2024)
- [Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness](https://mlsanity.com/paper/2305.19101/) (2023)
- [Detecting Out-Of-Distribution Earth Observation Images with Diffusion Models](https://mlsanity.com/paper/2404.12667/) (2024)
- [Accelerating Guided Diffusion Sampling with Splitting Numerical Methods](https://mlsanity.com/paper/2301.11558/) (2023)
- [Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback](https://mlsanity.com/paper/2307.02770/) (2023)
- [InstructGIE: Towards Generalizable Image Editing](https://mlsanity.com/paper/2403.05018/) (2024)
- [Pi-fusion: Physics-informed diffusion model for learning fluid dynamics](https://mlsanity.com/paper/2406.03711/) (2024)
- [Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models](https://mlsanity.com/paper/2410.09614/) (2024)
- [GeoGuide: Geometric guidance of diffusion models](https://mlsanity.com/paper/2407.12889/) (2024)
- [Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities](https://mlsanity.com/paper/2309.00952/) (2023)
- [Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning](https://mlsanity.com/paper/2606.11087/) (2026)
- [DiffAug: A Diffuse-and-Denoise Augmentation for Training Robust Classifiers](https://mlsanity.com/paper/2306.09192/) (2023)
- [Bootstrap Diffusion Model Curve Estimation for High Resolution Low-Light Image Enhancement](https://mlsanity.com/paper/2309.14709/) (2023)
- [Classifier Guidance Enhances Diffusion-based Adversarial Purification by Preserving Predictive Information](https://mlsanity.com/paper/2408.05900/) (2024)
- [Reward Sharpness-Aware Fine-Tuning for Diffusion Models](https://mlsanity.com/paper/2603.21175/) (2026)
- [Leveraging Graph Diffusion Models for Network Refinement Tasks](https://mlsanity.com/paper/2311.17856/) (2023)
- [ExpertGen: Training-Free Expert Guidance for Controllable Text-to-Face Generation](https://mlsanity.com/paper/2505.17256/) (2025)
- [Enhancing Diffusion Model Guidance through Calibration and Regularization](https://mlsanity.com/paper/2511.05844/) (2025)

## Abstract (arXiv)

Denoising diffusion probabilistic models (DDPMs) are a recent family of generative models that achieve state-of-the-art results. In order to obtain class-conditional generation, it was suggested to guide the diffusion process by gradients from a time-dependent classifier. While the idea is theoretically sound, deep learning-based classifiers are infamously susceptible to gradient-based adversarial attacks. Therefore, while traditional classifiers may achieve good accuracy scores, their gradients are possibly unreliable and might hinder the improvement of the generation results. Recent work discovered that adversarially robust classifiers exhibit gradients that are aligned with human perception, and these could better guide a generative process towards semantically meaningful images. We utilize this observation by defining and training a time-dependent adversarially robust classifier and use it as guidance for a generative diffusion model. In experiments on the highly challenging and diverse ImageNet dataset, our scheme introduces significantly more intelligible intermediate gradients, better alignment with theoretical findings, as well as improved generation results under several evaluation metrics. Furthermore, we conduct an opinion survey whose findings indicate that human raters prefer our method's results.