mlsanityData to 14 Oct 2026

Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners

arXiv 2305.10722 ↗First posted 18 May 2023cs.CVcited by 14

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation. Since text-to-image generation often requires models to generate visual concepts with fine-grained details and attributes specified in text prompts, can we leverage the powerful representations learned by pre-trained diffusion models for discriminative tasks such as image-text matching? To answer this question, we propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach mainly uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information and fine-tune the model via efficient attention-based prompt learning to perform image-text matching. By comparing DSD with state-of-the-art methods on several benchmark datasets, we demonstrate the potential of using pre-trained diffusion models for discriminative tasks with superior results on few-shot image-text matching.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 61 of its 72 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

61 of 72 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  2. 2023Your Diffusion Model is Secretly a Zero-Shot Classifier28 Mar
  3. 2023Text-to-Image Diffusion Models are Zero-Shot Classifiers27 Mar
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  5. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  6. 2022Fine-grained Image Editing by Pixel-wise Guidance Using Diffusion Models5 Dec
  7. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention10 Oct
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  12. 2022Visual Prompt Tuningcited by 1,681
  13. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  14. 2015Spatial Transformer Networkscited by 1,429
  15. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  16. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  17. 2016Modeling Context in Referring Expressionscited by 1,063
  18. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  19. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  20. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  21. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  22. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  23. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  24. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  25. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  26. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  27. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  28. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  29. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  30. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  31. 2018A Note on the Inception Scorecited by 226
  32. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  33. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 187
  34. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  35. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  36. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  37. 2023Diffusion Models as Masked Autoencoderscited by 71
  38. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 5; 2026: 2.

14 papers.

Sort:

By area:

  1. 2023Interpretable Diffusion via Information Decompositioncited by 37
  2. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 30
  3. 2023VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounderscited by 18
  4. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  5. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  6. 2024FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generationcited by 5
  7. 2024TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaborationcited by 5
  8. 2025Evaluating Compositional Generalisation in VLMs and Diffusion Modelscited by 5
  9. 2024Image and Video Quality Assessment using Prompt-Guided Latent Diffusion Models for Cross-Dataset Generalizationcited by 3
  10. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractorcited by 3
  11. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  12. 2025MorphoSim: An Interactive, Controllable, and Editable Language-guided 4D World Simulatorcited by 1
2 more
  1. 2026How Do Diffusion Classifiers Decide? A Bias-Centric Evaluationcited by 0
  2. 2026DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Are Diffusion Models Vision-And-Language Reasoners?
  2. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  3. 2023Discriminative Class Tokens for Text-to-Image Diffusion Models
  4. 2023Do text-free diffusion models learn discriminative visual representations?
  5. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception
  6. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
  7. 2023Harnessing Diffusion Models for Visual Perception with Meta Prompts
  8. 2023DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability