mlsanityData to 7 Oct 2026

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

arXiv 2303.04803 ↗First posted 8 Mar 2023cs.CVcited by 371

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Text-to-image diffusion models have the remarkable ability to generate high-quality images with diverse open-vocabulary language descriptions. This demonstrates that their internal representation space is highly correlated with open concepts in the real world. Text-image discriminative models like CLIP, on the other hand, are good at classifying images into open-vocabulary labels. We leverage the frozen internal representations of both these models to perform panoptic segmentation of any category in the wild. Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks. In particular, with COCO training only, our method achieves 23.4 PQ and 30.0 mIoU on the ADE20K dataset, with 8.3 PQ and 7.9 mIoU absolute improvement over the previous state of the art. We open-source our code and models at https://github.com/NVlabs/ODISE .

Key facts. Cited by 371 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 84 of its 105 references that are in the map. First posted on arXiv 8 Mar 2023.

Builds on

84 of 105 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec

Other references · 83

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2017Mask R-CNNcited by 7,259
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  13. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  14. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  15. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  16. 2022A ConvNet for the 2020scited by 3,545
  17. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  18. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  19. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  20. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  21. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  22. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  23. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  24. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  25. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  26. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  27. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  28. 2021Grounded Language-Image Pre-trainingcited by 1,147
  29. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  30. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  31. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  32. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  33. 2018Panoptic Segmentationcited by 796
  34. 2022Language-driven Semantic Segmentationcited by 711
  35. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  36. 2019Panoptic Feature Pyramid Networkscited by 644
  37. 2021Score-based Generative Modeling in Latent Spacecited by 634
  38. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 99; 2024: 147; 2025: 76; 2026: 49.

371 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2023Segment Everything Everywhere All at Oncecited by 546
  3. 2023Emergent Correspondence from Image Diffusioncited by 437
  4. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  5. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  6. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  7. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  8. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  9. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  10. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  11. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  12. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
38 more
  1. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  2. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  4. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  5. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  6. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  7. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  8. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  9. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  10. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 121
  11. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  12. 2023Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentationcited by 112
  13. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  14. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  15. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  16. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  17. 2023Diffusion Model as Representation Learnercited by 95
  18. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  19. 2023Diffusion Models for Imperceptible and Transferable Adversarial Attackcited by 93
  20. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 89
  21. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  22. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  23. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  24. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  25. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  26. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  27. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  28. 2024Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Modelcited by 73
  29. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  30. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  31. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  32. 2024UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peakscited by 64
  33. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  34. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  35. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  36. 2023Aligning and Prompting Everything All at Once for Universal Visual Perceptioncited by 59
  37. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  38. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54

50 most cited of 371 All 371 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Open-Vocabulary Panoptic Segmentation Using BERT Pre-Training of Vision-Language Multiway Transformer Model
  2. 2023Open-vocabulary Object Segmentation with Diffusion Models
  3. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
  4. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  5. 2024Panoptic Diffusion Models: co-generation of images and segmentation maps
  6. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model
  7. 2026MAVISEG: Manifold Propagation and Visual Prototypes for Zero-Shot Open-Vocabulary Segmentation in Diffusion Transformers
  8. 2025NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentation