mlsanityData to 7 Oct 2026

Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion

arXiv 2308.12469 ↗First posted 23 Aug 2023cs.CVcited by 131

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Producing quality segmentation masks for images is a fundamental problem in computer vision. Recent research has explored large-scale supervised training to enable zero-shot segmentation on virtually any image style and unsupervised training to enable segmentation without dense annotations. However, constructing a model capable of segmenting anything in a zero-shot manner without any annotations is still challenging. In this paper, we propose to utilize the self-attention layers in stable diffusion models to achieve this goal because the pre-trained stable diffusion model has learned inherent concepts of objects within its attention layers. Specifically, we introduce a simple yet effective iterative merging process based on measuring KL divergence among attention maps to merge them into valid segmentation masks. The proposed method does not require any training or language dependency to extract quality segmentation for any images. On COCO-Stuff-27, our method surpasses the prior unsupervised zero-shot SOTA method by an absolute 26% in pixel accuracy and 17% in mean IoU. The project page is at urlhttps://sites.google.com/view/diffseg/home.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 35 of its 47 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

35 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models21 Mar
  2. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  3. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec
  4. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Models1 Dec

Other references · 31

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Segment Anythingcited by 8,158
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
19 more
  1. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  2. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  5. 2021Image Segmentation Using Text and Image Promptscited by 509
  6. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  7. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  8. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  9. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  10. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  11. 2019Zero-Shot Semantic Segmentationcited by 245
  12. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  13. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  14. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  15. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  16. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  17. 2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
  18. 2022ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentationcited by 42
  19. 2022NamedMask: Distilling Segmenters from Complementary Foundation Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 40; 2025: 52; 2026: 26.

131 papers.

Sort:

By area:

  1. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  2. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 153
  3. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 85
  4. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
  5. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 48
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  7. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  8. 2023Interpretable Diffusion via Information Decompositioncited by 37
  9. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 37
  10. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  11. 2023SLiMe: Segment Like Mecited by 34
  12. 2024DragAPart: Learning a Part-Level Motion Prior for Articulated Objectscited by 32
38 more
  1. 2023UniGS: Unified Representation for Image Generation and Segmentationcited by 29
  2. 2024Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Modelscited by 29
  3. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  4. 2024Video Diffusion Models: A Surveycited by 28
  5. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  6. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformerscited by 24
  7. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 23
  8. 2023MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Modelcited by 18
  9. 2024Where's Waldo: Diffusion Features for Personalized Segmentation and Retrievalcited by 18
  10. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  11. 2025TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generationcited by 16
  12. 2023Unsupervised Keypoints from Pretrained Diffusion Modelscited by 15
  13. 2024DP-IQA: Utilizing Diffusion Prior for Blind Image Quality Assessment in the Wildcited by 14
  14. 2025On Denoising Walking Videos for Gait Recognitioncited by 14
  15. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  16. 2024Object Pose Estimation via the Aggregation of Diffusion Featurescited by 12
  17. 2024Augmented Object Intelligence with XR-Objectscited by 12
  18. 2024MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentationcited by 11
  19. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11
  20. 2025PixCell: A generative foundation model for digital histopathology imagescited by 11
  21. 2025Attention (as Discrete-Time Markov) Chainscited by 11
  22. 2023Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentationcited by 10
  23. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  24. 2025Can Generative Geospatial Diffusion Models Excel as Discriminative Geospatial Foundation Models?cited by 10
  25. 2025Scene-Centric Unsupervised Panoptic Segmentationcited by 10
  26. 2024Fast constrained sampling in pre-trained diffusion modelscited by 9
  27. 2024Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentationcited by 9
  28. 2024Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognitioncited by 9
  29. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 8
  30. 2025Diffusion Counterfactual Generation with Semantic Abductioncited by 8
  31. 2025Stable Diffusion Models are Secretly Good at Visual In-Context Learningcited by 8
  32. 2024AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customizationcited by 6
  33. 2024ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Groundingcited by 6
  34. 2024Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generationcited by 6
  35. 2025MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matchingcited by 6
  36. 2025Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Modelscited by 6
  37. 2024Bayesian Active Learning for Semantic Segmentationcited by 5
  38. 2025Few-Shot Adaptation of Training-Free Foundation Model for 3D Medical Image Segmentationcited by 5

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
  2. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation
  3. 2026Prompting Diffusion Models for Zero-Shot Instance Segmentation
  4. 2025Studying Image Diffusion Features for Zero-Shot Video Object Segmentation
  5. 2026Making Training-Free Diffusion Segmentors Scale with the Generative Power
  6. 2025Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
  7. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
  8. 2023Masked Diffusion as Self-supervised Representation Learner