mlsanityData to 7 Oct 2026

Diffusion Models for Open-Vocabulary Segmentation

arXiv 2306.09316 ↗First posted 15 Jun 2023cs.CVcited by 63

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Open-vocabulary segmentation is the task of segmenting anything that can be named in an image. Recently, large-scale vision-language modelling has led to significant advances in open-vocabulary segmentation, but at the cost of gargantuan and increasing training and annotation efforts. Hence, we ask if it is possible to use existing foundation models to synthesise on-demand efficient segmentation algorithms for specific class sets, making them applicable in an open-vocabulary setting without the need to collect further data, annotations or perform training. To that end, we present OVDiff, a novel method that leverages generative text-to-image diffusion models for unsupervised open-vocabulary segmentation. OVDiff synthesises support image sets for arbitrary textual categories, creating for each a set of prototypes representative of both the category and its surrounding context (background). It relies solely on pre-trained components and outputs the synthesised segmenter directly, without training. Our approach shows strong performance on a range of benchmarks, obtaining a lead of more than 5% over prior work on PASCAL VOC.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 70 of its 87 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

70 of 87 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models21 Mar
  2. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  3. 2023Open-vocabulary Object Segmentation with Diffusion Models12 Jan
  4. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  7. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  8. 2022Language-driven Semantic Segmentationcited by 711
  9. 2021Extract Free Dense Labels from CLIPcited by 580
  10. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  12. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  13. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  14. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  15. 2019Zero-Shot Semantic Segmentationcited by 245
  16. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  17. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  18. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  19. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  20. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  21. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 186
  22. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  23. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
  24. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  25. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  26. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  27. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  28. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  29. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  30. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  31. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  32. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  33. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  34. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  35. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  36. 2019Unsupervised Object Segmentation by Redrawingcited by 74
  37. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  38. 2022Unsupervised Object Localization: Observing the Background to Discover Objectscited by 66

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 19; 2025: 17; 2026: 11.

63 papers.

Sort:

By area:

  1. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  3. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  4. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  5. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  6. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  7. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  8. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
  9. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 48
  10. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 47
  11. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 43
  12. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
38 more
  1. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  2. 2023Interpretable Diffusion via Information Decompositioncited by 37
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 33
  5. 2024DragAPart: Learning a Part-Level Motion Prior for Articulated Objectscited by 32
  6. 2023Tokenize Anything via Promptingcited by 31
  7. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformerscited by 24
  8. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  9. 2024Alfie: Democratising RGBA Image Generation With No cited by 20
  10. 2023VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounderscited by 18
  11. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  12. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  13. 2024Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restorationcited by 15
  14. 2023AI-Generated Images as Data Source: The Dawn of Synthetic Eracited by 13
  15. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  16. 2024Placing Objects in Context via Inpainting for Out-of-distribution Segmentationcited by 12
  17. 2023Zero-Shot Object Counting with Language-Vision Modelscited by 9
  18. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  19. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  20. 2023OV-VG: A Benchmark for Open-Vocabulary Visual Groundingcited by 7
  21. 2023Exploring Social Bias in Downstream Applications of Text-to-Image Foundation Modelscited by 7
  22. 2024AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentationcited by 5
  23. 2024XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentationcited by 5
  24. 2025Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detectioncited by 5
  25. 2025S3OD: Towards Generalizable Salient Object Detection with Synthetic Datacited by 5
  26. 2023Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignmentcited by 4
  27. 2024Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priorscited by 4
  28. 2025Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentationcited by 4
  29. 2025Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scalingcited by 4
  30. 2025What Holds Back Open-Vocabulary Segmentation?cited by 4
  31. 2024Mapping the Mind of an Instruction-based Image Editing using SMILEcited by 3
  32. 2025Superpowering Open-Vocabulary Object Detectors for X-ray Visioncited by 3
  33. 2025VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labelscited by 3
  34. 2024SA-MixNet: Structure-aware Mixup and Invariance Learning for Scribble-supervised Road Extraction in Remote Sensing Imagescited by 2
  35. 2025DIVE: Inverting Conditional Diffusion Models for Discriminative Taskscited by 2
  36. 2026AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectorscited by 2
  37. 2026ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentationcited by 2
  38. 2024Outline-Guided Object Inpainting with Diffusion Modelscited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Transferable and Principled Efficiency for Open-Vocabulary Segmentation
  2. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  3. 2024Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models
  4. 2026dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
  5. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  6. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation
  7. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
  8. 2025From Open-Vocabulary to Vocabulary-Free Semantic Segmentation