mlsanityData to 14 Oct 2026

Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors

arXiv 2211.13224 ↗First posted 23 Nov 2022cs.CV cs.CL cs.LGcited by 34

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Recently, text-to-image diffusion models have shown remarkable capabilities in creating realistic images from natural language prompts. However, few works have explored using these models for semantic localization or grounding. In this work, we explore how an off-the-shelf text-to-image diffusion model, trained without exposure to localization information, can ground various semantic phrases without segmentation-specific re-training. We introduce an inference time optimization process capable of generating segmentation masks conditioned on natural language prompts. Our proposal, Peekaboo, is a first-of-its-kind zero-shot, open-vocabulary, unsupervised semantic grounding technique leveraging diffusion models without any training. We evaluate Peekaboo on the Pascal VOC dataset for unsupervised semantic segmentation and the RefCOCO dataset for referring segmentation, showing results competitive with promising results. We also demonstrate how Peekaboo can be used to generate images with transparency, even though the underlying diffusion model was only trained on RGB images - which to our knowledge we are the first to attempt. Please see our project page, including our code: https://ryanndagreat.github.io/peekaboo

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 69 of its 90 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

69 of 90 references.

References · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  12. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  13. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  14. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  15. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  16. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  18. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  19. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  20. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  21. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  22. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  24. 2022Language-driven Semantic Segmentationcited by 717
  25. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  26. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  27. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  28. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  29. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  30. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  31. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
  32. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  33. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  34. 2021Advances in Neural Renderingcited by 365
  35. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  36. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  37. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  38. 2016Segmentation from Natural Language Expressionscited by 257

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 17; 2024: 10; 2025: 7.

34 papers.

Sort:

By area:

  1. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  2. 2025Video models are zero-shot learners and reasonerscited by 220
  3. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  4. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  5. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  6. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  7. 2023Diffusion Models Beat GANs on Image Classificationcited by 58
  8. 2023Intriguing properties of generative classifierscited by 52
  9. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 40
  10. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  11. 2023SLiMe: Segment Like Mecited by 34
  12. 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
22 more
  1. 2023UniGS: Unified Representation for Image Generation and Segmentationcited by 29
  2. 2023Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedbackcited by 27
  3. 2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26
  4. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  5. 2025Multimodal Referring Segmentation: A Surveycited by 22
  6. 2024DeBaRA: Denoising-Based 3D Room Arrangement Generationcited by 20
  7. 2023MDP: A Generalized Framework for Text-Guided Image Editing by Manipulating the Diffusion Pathcited by 18
  8. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  9. 2023Denoising Diffusion Semantic Segmentation with Mask Prior Modelingcited by 15
  10. 2023Are Diffusion Models Vision-And-Language Reasoners?cited by 14
  11. 2024Matting by Generationcited by 13
  12. 2023Gen2Det: Generate to Detectcited by 12
  13. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  14. 2024Chronologically Accurate Retrieval for Temporal Grounding of Motion-Language Modelscited by 12
  15. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  16. 2025Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulationcited by 11
  17. 2024Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognitioncited by 9
  18. 2024A Simple and Efficient Baseline for Zero-Shot Generative Classificationcited by 7
  19. 2025G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Modelscited by 5
  20. 2024Towards Generative Class Prompt Learning for Fine-grained Visual Recognitioncited by 2
  21. 2025Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptationcited by 2
  22. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimizationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Open-vocabulary Object Segmentation with Diffusion Models
  2. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model
  3. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentation
  4. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
  5. 2026Diffusion Model as a Generalist Segmentation Learner
  6. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
  7. 2023VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders
  8. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation