mlsanityData to 14 Oct 2026

Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models

arXiv 2308.16777 ↗First posted 31 Aug 2023cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

Zero-shot referring image segmentation is a challenging task because it aims to find an instance segmentation mask based on the given referring descriptions, without training on this type of paired data. Current zero-shot methods mainly focus on using pre-trained discriminative models (e.g., CLIP). However, we have observed that generative models (e.g., Stable Diffusion) have potentially understood the relationships between various visual elements and text descriptions, which are rarely investigated in this task. In this work, we introduce a novel Referring Diffusional segmentor (Ref-Diff) for this task, which leverages the fine-grained multi-modal information from generative models. We demonstrate that without a proposal generator, a generative model alone can achieve comparable performance to existing SOTA weakly-supervised models. When we combine both generative and discriminative models, our Ref-Diff outperforms these competing methods by a significant margin. This indicates that generative models are also beneficial for this task and can complement discriminative models for better referring segmentation. Our code is publicly available at https://github.com/kodenii/Ref-Diff.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

39 of 50 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Segment Anythingcited by 8,234
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  11. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  12. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
27 more
  1. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  2. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  3. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  4. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  5. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  6. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 280
  7. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  8. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  9. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  10. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  11. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  12. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  13. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  14. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  15. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  16. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  17. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 111
  18. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  19. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  20. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  21. 2023Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentationcited by 36
  22. 2023Parallel Vertex Diffusion for Unified Visual Groundingcited by 28
  23. 2022Weakly-supervised segmentation of referring expressionscited by 21
  24. 2022Decoder Denoising Pretraining for Semantic Segmentationcited by 20
  25. 2023Meta Compositional Referring Expression Segmentationcited by 18
  26. 2023Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentationcited by 16
  27. 2023Co-attention Propagation Network for Zero-Shot Video Object Segmentationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 11; 2025: 11; 2026: 7.

33 papers.

Sort:

By area:

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  2. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2024Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Modelscited by 29
  5. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  6. 2024MC2: Multi-concept Guidance for Customized Multi-concept Generationcited by 26
  7. 2025IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasiscited by 26
  8. 2023Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentationcited by 22
  9. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  10. 2024Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoningcited by 17
  11. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  12. 2024VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Modelscited by 14
21 more
  1. 2025Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentationcited by 14
  2. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  3. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  4. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 9
  5. 2025JOG3R: Towards 3D-Consistent Video Generatorscited by 9
  6. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  7. 2024Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniquescited by 2
  8. 2025GS: Generative Segmentation via Label Diffusioncited by 2
  9. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentationcited by 2
  10. 2025In-Situ Tweedie Discrete Diffusion Modelscited by 2
  11. 2025ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Modelscited by 1
  12. 2025Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Reviewcited by 1
  13. 2026AR2-4FV: Anchored Referring and Re-identification for Long-Term Grounding in Fixed-View Videoscited by 1
  14. 2026RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulationcited by 1
  15. 2026Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentationcited by 1
  16. 2026Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentationcited by 1
  17. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  18. 2023ResEnsemble-DDPM: Residual Denoising Diffusion Probabilistic Models for Ensemble Learningcited by 0
  19. 2025DGL-RSIS: Decoupling Global Spatial Context and Local Class Semantics for Training-Free Remote Sensing Image Segmentationcited by 0
  20. 2026Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentationcited by 0
  21. 2026Prompting Diffusion Models for Zero-Shot Instance Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentation
  2. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  3. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
  4. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors
  5. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
  6. 2026Making Training-Free Diffusion Segmentors Scale with the Generative Power
  7. 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
  8. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Models