mlsanityData to 7 Oct 2026

DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery

arXiv 2303.09813 ↗First posted 17 Mar 2023cs.CV cs.AIcited by 53

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Learning from a large corpus of data, pre-trained models have achieved impressive progress nowadays. As popular generative pre-training, diffusion models capture both low-level visual knowledge and high-level semantic relations. In this paper, we propose to exploit such knowledgeable diffusion models for mainstream discriminative tasks, i.e., unsupervised object discovery: saliency segmentation and object localization. However, the challenges exist as there is one structural difference between generative and discriminative models, which limits the direct use. Besides, the lack of explicitly labeled data significantly limits performance in unsupervised settings. To tackle these issues, we introduce DiffusionSeg, one novel synthesis-exploitation framework containing two-stage strategies. To alleviate data insufficiency, we synthesize abundant images, and propose a novel training-free AttentionCut to obtain masks in the first synthesis stage. In the second exploitation stage, to bridge the structural gap, we use the inversion technique, to map the given image back to diffusion features. These features can be directly used by downstream architectures. Extensive experiments and ablation studies demonstrate the superiority of adapting diffusion for unsupervised object discovery.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 61 of its 86 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

61 of 86 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  4. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  10. 2014Conditional Generative Adversarial Netscited by 2,152
  11. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  12. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  13. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  14. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  15. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  16. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  17. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  18. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  19. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  20. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  21. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  22. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  23. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  24. 2016Deeply supervised salient object detection with short connectionscited by 196
  25. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
  26. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  27. 2019Pyramid Feature Attention Network for Saliency detectioncited by 128
  28. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  29. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  30. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  31. 2015The Fast Bilateral Solvercited by 86
  32. 2019Unsupervised Object Segmentation by Redrawingcited by 74
  33. 2022Unsupervised Object Localization: Observing the Background to Discover Objectscited by 66
  34. 2020Toward unsupervised, multi-object discovery in large-scale image collectionscited by 65
  35. 2019Emergence of Object Segmentation in Perturbed Generative Modelscited by 57
  36. 2022Unsupervised Salient Object Detection with Spectral Cluster Votingcited by 56
  37. 2021Large-Scale Unsupervised Object Discoverycited by 54
  38. 2020Object Segmentation Without Labels with Large-Scale Generative Modelscited by 52

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 16; 2025: 15; 2026: 8.

53 papers.

Sort:

By area:

  1. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 121
  2. 2023Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentationcited by 112
  3. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  4. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  5. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  6. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  7. 2023Interpretable Diffusion via Information Decompositioncited by 37
  8. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  9. 2023AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregationcited by 29
  10. 2023UniGS: Unified Representation for Image Generation and Segmentationcited by 29
  11. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  12. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
38 more
  1. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  2. 2023Multi-modal Prompting for Low-Shot Temporal Action Localizationcited by 19
  3. 2023Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Surveycited by 16
  4. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  5. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  6. 2024GenDeg: Diffusion-based Degradation Synthesis for Generalizable All-In-One Image Restorationcited by 13
  7. 2024Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-trainingcited by 13
  8. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  9. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Modelscited by 11
  10. 2023Zero-Shot Object Counting with Language-Vision Modelscited by 9
  11. 2024DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognitioncited by 8
  12. 2025Decouple before Align: Visual Disentanglement Enhances Prompt Tuningcited by 7
  13. 2025G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Modelscited by 5
  14. 2025SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Modelscited by 5
  15. 2023Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentationcited by 4
  16. 2024GRIF-DM: Generation of Rich Impression Fonts using Diffusion Modelscited by 4
  17. 2024Training-Free Layout-to-Image Generation with Marginal Attention Constraintscited by 4
  18. 2025MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognitioncited by 4
  19. 2024Image and Video Quality Assessment using Prompt-Guided Latent Diffusion Models for Cross-Dataset Generalizationcited by 3
  20. 2025ConText: Driving In-context Learning for Text Removal and Segmentationcited by 3
  21. 2025SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentationcited by 3
  22. 2025Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimizationcited by 3
  23. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3
  24. 2024Free-Mask: A Novel Paradigm of Integration Between the Segmentation Diffusion Model and Image Editingcited by 2
  25. 2024No Annotations for Object Detection in Art through Stable Diffusioncited by 2
  26. 2025Squeeze Out Tokens from Sample for Finer-Grained Data Governancecited by 2
  27. 2025Marine Saliency Segmenter: Object-Focused Conditional Diffusion with Region-Level Semantic Knowledge Distillationcited by 2
  28. 2025The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Modelscited by 2
  29. 2025GS: Generative Segmentation via Label Diffusioncited by 2
  30. 2025In-Situ Tweedie Discrete Diffusion Modelscited by 2
  31. 2026Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoningcited by 2
  32. 2024Mitigate Target-level Insensitivity of Infrared Small Target Detection via Posterior Distribution Modelingcited by 1
  33. 2024Evaluating Text-to-Image Diffusion Models for Texturing Synthetic Datacited by 1
  34. 2025Leveraging Prior Knowledge of Diffusion Model for Person Searchcited by 1
  35. 2025Wave-Particle (Continuous-Discrete) Dualistic Visual Tokenization for Unified Understanding and Generationcited by 1
  36. 2026Dynamic Training-Free Fusion of Subject and Style LoRAscited by 0
  37. 2026TransText: Alpha-as-RGB Representation for Transparent Text Animationcited by 0
  38. 2026MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generationcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Do text-free diffusion models learn discriminative visual representations?
  2. 2025DIVE: Inverting Conditional Diffusion Models for Discriminative Tasks
  3. 2023Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks
  4. 2024Diffusion Models in Low-Level Vision: A Survey
  5. 2025Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detection
  6. 2026Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision
  7. 2026Diffusion Model as a Generalist Segmentation Learner
  8. 2026Making Training-Free Diffusion Segmentors Scale with the Generative Power