mlsanityData to 7 Oct 2026

Label-Efficient Semantic Segmentation with Diffusion Models

arXiv 2112.03126 ↗First posted 6 Dec 2021cs.CV cs.LGcited by 495

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision.

Key facts. Cited by 495 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 36 of its 39 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

36 of 39 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
24 more
  1. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  5. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  6. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  8. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  9. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  10. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  11. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  12. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  13. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  14. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  15. 2019Large Scale Adversarial Representation Learningcited by 238
  16. 2020Adversarial Latent Autoencoderscited by 155
  17. 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
  18. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  19. 2021Repurposing GANs for One-shot Semantic Part Segmentationcited by 88
  20. 2020Generative Hierarchical Features from Synthesizing Imagescited by 83
  21. 2020Object Segmentation Without Labels with Large-Scale Generative Modelscited by 52
  22. 2021Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Modelscited by 44
  23. 2021Linear Semantics in Generative Adversarial Networkscited by 25
  24. 2020Learning High-Resolution Domain-Specific Representations with a GAN Generatorcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 35; 2023: 172; 2024: 147; 2025: 105; 2026: 36.

495 papers.

Sort:

By area:

  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  3. 2022Diffusion Models in Vision: A Surveycited by 1,008
  4. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  5. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  6. 2023Emergent Correspondence from Image Diffusioncited by 437
  7. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  8. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  9. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  10. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  11. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  12. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
38 more
  1. 2022A Survey on Generative Diffusion Modelcited by 238
  2. 2022Diffusion Models for Medical Anomaly Detectioncited by 231
  3. 2022Diffusion models as plug-and-play priorscited by 230
  4. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 216
  5. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  6. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  7. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  8. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  9. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  10. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  11. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  12. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 121
  13. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  14. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  15. 2023MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusioncited by 115
  16. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 113
  17. 2023Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentationcited by 112
  18. 2022How to Backdoor Diffusion Models?cited by 102
  19. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  20. 2023Diffusion Model as Representation Learnercited by 95
  21. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  22. 2023Ambiguous Medical Image Segmentation using Diffusion Modelscited by 92
  23. 2023Object-Centric Slot Diffusioncited by 90
  24. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  25. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  26. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  27. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 81
  28. 2022Diffusion Models for Counterfactual Explanationscited by 79
  29. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  30. 2023Diffusion Action Segmentationcited by 74
  31. 2024Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Modelcited by 73
  32. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  33. 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Modelscited by 68
  34. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  35. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 66
  36. 2024UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peakscited by 64
  37. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  38. 2023Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfercited by 59

50 most cited of 495 All 495 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation
  2. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
  3. 2021Diffusion Models for Implicit Image Segmentation Ensembles
  4. 2023Prompting Diffusion Representations for Cross-Domain Semantic Segmentation
  5. 2025GS: Generative Segmentation via Label Diffusion
  6. 2024Diffusion Models in Low-Level Vision: A Survey
  7. 2023Diffusion-based Image Translation with Label Guidance for Domain Adaptive Semantic Segmentation
  8. 2023Open-vocabulary Object Segmentation with Diffusion Models