mlsanityData to 7 Oct 2026

Open-vocabulary Object Segmentation with Diffusion Models

arXiv 2301.05221 ↗First posted 12 Jan 2023cs.CVcited by 82

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

The goal of this paper is to extract the visual-language correspondence from a pre-trained text-to-image diffusion model, in the form of segmentation map, i.e., simultaneously generating images and segmentation masks for the corresponding visual entities described in the text prompt. We make the following contributions: (i) we pair the existing Stable Diffusion model with a novel grounding module, that can be trained to align the visual and textual embedding space of the diffusion model with only a small number of object categories; (ii) we establish an automatic pipeline for constructing a dataset, that consists of {image, segmentation mask, text prompt} triplets, to train the proposed grounding module; (iii) we evaluate the performance of open-vocabulary grounding on images generated from the text-to-image diffusion model and show that the module can well segment the objects of categories beyond seen ones at training time; (iv) we adopt the augmented diffusion model to build a synthetic semantic segmentation dataset, and show that, training a standard segmentation model on such dataset demonstrates competitive performance on the zero-shot segmentation(ZS3) benchmark, which opens up new opportunities for adopting the powerful diffusion model for discriminative tasks.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 38 of its 42 references that are in the map. First posted on arXiv 12 Jan 2023.

Builds on

38 of 42 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Generative Adversarial Networkscited by 5,603
26 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  4. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  5. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  10. 2019Zero-Shot Semantic Segmentationcited by 245
  11. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  12. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  13. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  14. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  15. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  16. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  17. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  18. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  19. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  20. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  21. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  22. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  23. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  24. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  25. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  26. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 38; 2025: 23; 2026: 6.

82 papers.

Sort:

By area:

  1. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  2. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 153
  3. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  4. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  5. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  6. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  7. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  8. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  9. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  10. 2023GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generationcited by 56
  11. 2023StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generationcited by 56
  12. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 48
38 more
  1. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 43
  2. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
  3. 2024One Diffusion to Generate Them Allcited by 41
  4. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  5. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  6. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  7. 2025UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Modelscited by 38
  8. 2023A General Protocol to Probe Large Vision Models for 3D Physical Understandingcited by 32
  9. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  10. 2024Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Modelscited by 29
  11. 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentationcited by 27
  12. 2024InstaGen: Enhancing Object Detection by Training on Synthetic Datasetcited by 26
  13. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  14. 2024Alfie: Democratising RGBA Image Generation With No cited by 20
  15. 2023SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesiscited by 18
  16. 2023MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Maskcited by 17
  17. 2024ODGEN: Domain-specific Object Detection Data Generation with Diffusion Modelscited by 17
  18. 2024Open Panoramic Segmentationcited by 17
  19. 2025BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidancecited by 17
  20. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  21. 2024DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Datacited by 15
  22. 2023AI-Generated Images as Data Source: The Dawn of Synthetic Eracited by 13
  23. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  24. 2025Unified Dense Prediction of Video Diffusioncited by 13
  25. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  26. 2024LAKE-RED: Camouflaged Images Generation by Latent Background Knowledge Retrieval-Augmented Diffusioncited by 12
  27. 2024Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Modelscited by 12
  28. 2024Generative Active Learning for Long-tailed Instance Segmentationcited by 10
  29. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  30. 2024OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generationcited by 10
  31. 2023Guided Diffusion from Self-Supervised Diffusion Featurescited by 9
  32. 2025Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Methodcited by 8
  33. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  34. 2024SynFER: Towards Boosting Facial Expression Recognition with Synthetic Datacited by 6
  35. 2023ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentationcited by 5
  36. 2024Generative Edge Detection with Stable Diffusioncited by 5
  37. 2025G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Modelscited by 5
  38. 2025SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Modelscited by 5

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  2. 2025FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation
  3. 2026Diffusion Model as a Generalist Segmentation Learner
  4. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors
  5. 2026Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision
  6. 2024MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentation
  7. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  8. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers