mlsanityData to 7 Oct 2026

Your Diffusion Model is Secretly a Zero-Shot Classifier

arXiv 2303.16203 ↗First posted 28 Mar 2023cs.LG cs.AI cs.CVcited by 301

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density estimates, which are useful for tasks beyond image generation. In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. Although a gap remains between generative and discriminative approaches on zero-shot recognition tasks, our diffusion-based approach has significantly stronger multimodal compositional reasoning ability than competing discriminative approaches. Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Our models achieve strong classification performance using only weak augmentations and exhibit qualitatively better "effective robustness" to distribution shift. Overall, our results are a step toward using generative over discriminative models for downstream tasks. Results and visualizations at https://diffusion-classifier.github.io/

Key facts. Cited by 301 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 61 of its 89 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

61 of 89 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  5. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  13. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  14. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  15. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  16. 2016Density estimation using Real NVPcited by 2,029
  17. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  18. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  19. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  20. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  21. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  22. 2016Variational Inference: A Review for Statisticianscited by 1,466
  23. 2019Natural Adversarial Examplescited by 1,401
  24. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  25. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  26. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  27. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  28. 2021Grounded Language-Image Pre-trainingcited by 1,147
  29. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  30. 2016Pixel Recurrent Neural Networkscited by 814
  31. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  32. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  33. 2018Do Better ImageNet Models Transfer Better?cited by 560
  34. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  35. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  36. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  37. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  38. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 44; 2024: 108; 2025: 96; 2026: 53.

301 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  3. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  5. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  6. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  7. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  8. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 153
  9. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  10. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 89
  11. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  12. 2023Robust Classification via a Single Diffusion Modelcited by 79
38 more
  1. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  2. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 76
  3. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  4. 2024Analysis of Classifier-Free Guidance Weight Schedulerscited by 64
  5. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  6. 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 56
  7. 2023An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initializationcited by 53
  8. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  9. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 48
  10. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 47
  11. 2025Unified Multimodal Discrete Diffusioncited by 46
  12. 2023Evaluating Data Attribution for Text-to-Image Modelscited by 45
  13. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  14. 2023MobileDiffusion: Instant Text-to-Image Generation on Mobile Devicescited by 42
  15. 2024CleanDIFT: Diffusion Features without Noisecited by 42
  16. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  17. 2024Score Distillation via Reparametrized DDIMcited by 41
  18. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  19. 2026Image Generators are Generalist Vision Learnerscited by 40
  20. 2024Neural Network Diffusioncited by 39
  21. 2024Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancycited by 38
  22. 2023Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Provingcited by 37
  23. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  24. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  25. 2023Fill-Up: Balancing Long-Tailed Data with Generative Modelscited by 34
  26. 2024Enhance Image Classification via Inter-Class Image Mixup with Diffusion Modelcited by 34
  27. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  28. 2024Simple ReFlow: Improved Techniques for Fast Flow Modelscited by 34
  29. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  30. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  31. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  32. 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
  33. 2023IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AIcited by 32
  34. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  35. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  36. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  37. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
  38. 2024Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulationcited by 29

50 most cited of 301 All 301 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-to-Image Diffusion Models are Zero-Shot Classifiers
  2. 2024A Simple and Efficient Baseline for Zero-Shot Generative Classification
  3. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification
  4. 2023Discriminative Class Tokens for Text-to-Image Diffusion Models
  5. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
  6. 2024Plug-and-Play Diffusion Distillation
  7. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
  8. 2024Diffusion Models Need Visual Priors for Image Generation