mlsanityData to 7 Oct 2026

Text-to-Image Diffusion Models are Zero-Shot Classifiers

arXiv 2303.15233 ↗First posted 27 Mar 2023cs.CV cs.AI cs.LGcited by 140

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data. However, what knowledge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks. We investigate diffusion models by proposing a method for evaluating them as zero-shot classifiers. The key idea is using a diffusion model's ability to denoise a noised image given a text description of a label as a proxy for that label's likelihood. We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge and comparing them with CLIP's zero-shot abilities. They perform competitively with CLIP on a wide range of zero-shot image classification datasets. Additionally, they achieve state-of-the-art results on shape/texture bias tests and can successfully perform attribute binding while CLIP cannot. Although generative pre-training is prevalent in NLP, visual foundation models often use other methods such as contrastive learning. Based on our findings, we argue that generative pre-training should be explored as a compelling alternative for vision-language tasks.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 38 of its 54 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

38 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  2. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2017On Calibration of Modern Neural Networkscited by 4,860
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
20 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  8. 2021Variational Diffusion Modelscited by 1,145
  9. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  10. 2021Scaling Vision Transformerscited by 993
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  13. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  14. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  15. 2022Diffusion Models for Adversarial Purificationcited by 469
  16. 2022Visual Prompting via Image Inpaintingcited by 248
  17. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  18. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  19. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentorscited by 34
  20. 2021CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classificationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 52; 2025: 37; 2026: 31.

140 papers.

Sort:

By area:

  1. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  2. 2025Video models are zero-shot learners and reasonerscited by 208
  3. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  4. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  5. 2023Diffusion Models for Imperceptible and Transferable Adversarial Attackcited by 93
  6. 2023Robust Classification via a Single Diffusion Modelcited by 79
  7. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 76
  8. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  9. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
  10. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  11. 2023Intriguing properties of generative classifierscited by 52
  12. 2024LaRE2: Latent Reconstruction Error Based Method for Diffusion-Generated Image Detectioncited by 47
38 more
  1. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  2. 2026Image Generators are Generalist Vision Learnerscited by 40
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  5. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  6. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  7. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 29
  8. 2023Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedbackcited by 27
  9. 2024Fast, Scale-Adaptive, and Uncertainty-Aware Downscaling of Earth System Model Fields with Generative Machine Learningcited by 26
  10. 2024Your Diffusion Model is Secretly a Certifiably Robust Classifiercited by 24
  11. 2024TextToon: Real-Time Text Toonify Head Avatar from Single Videocited by 21
  12. 2024DeBaRA: Denoising-Based 3D Room Arrangement Generationcited by 20
  13. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19
  14. 2023Unsupervised Keypoints from Pretrained Diffusion Modelscited by 15
  15. 2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learnerscited by 14
  16. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  17. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  18. 2025On Denoising Walking Videos for Gait Recognitioncited by 14
  19. 2023Gen2Det: Generate to Detectcited by 12
  20. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  21. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  22. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  23. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  24. 2024MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentationcited by 11
  25. 2024GenRec: Unifying Video Generation and Recognition with Diffusion Modelscited by 11
  26. 2024Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptorscited by 10
  27. 2024ZeroDiff: Solidified Visual-Semantic Correlation in Zero-Shot Learningcited by 10
  28. 2024Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifiercited by 10
  29. 2024Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognitioncited by 9
  30. 2025Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Freecited by 9
  31. 2025Generate, Refine, and Encode: Leveraging Synthesized Novel Samples for On-the-Fly Fine-Grained Category Discoverycited by 9
  32. 2024Preference Diffusion for Recommendationcited by 8
  33. 2025Diffusion Counterfactual Generation with Semantic Abductioncited by 8
  34. 2025Multimodal Atmospheric Super-Resolution With Deep Generative Modelscited by 8
  35. 2024Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Modelscited by 7
  36. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  37. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  38. 2025Noise Matters: Optimizing Matching Noise for Diffusion Classifierscited by 7

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Your Diffusion Model is Secretly a Zero-Shot Classifier
  2. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification
  3. 2023Discriminative Class Tokens for Text-to-Image Diffusion Models
  4. 2024A Simple and Efficient Baseline for Zero-Shot Generative Classification
  5. 2023Are Diffusion Models Vision-And-Language Reasoners?
  6. 2023Diversity and Diffusion: Observations on Synthetic Image Distributions with Stable Diffusion
  7. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decomposition
  8. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion