mlsanityData to 14 Oct 2026

CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free

arXiv 2309.14289 ↗First posted 25 Sep 2023cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

The emergence of CLIP has opened the way for open-world image perception. The zero-shot classification capabilities of the model are impressive but are harder to use for dense tasks such as image segmentation. Several methods have proposed different modifications and learning schemes to produce dense output. Instead, we propose in this work an open-vocabulary semantic segmentation method, dubbed CLIP-DIY, which does not require any additional training or annotations, but instead leverages existing unsupervised object localization approaches. In particular, CLIP-DIY is a multi-scale approach that directly exploits CLIP classification abilities on patches of different sizes and aggregates the decision in a single map. We further guide the segmentation using foreground/background scores obtained using unsupervised object localization methods. With our method, we obtain state-of-the-art zero-shot semantic segmentation results on PASCAL VOC and perform on par with the best methods on COCO. The code is available at http://github.com/wysoczanska/clip-diy

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

49 of 65 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
37 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  4. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  6. 2021Extract Free Dense Labels from CLIPcited by 583
  7. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  8. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2019Zero-Shot Semantic Segmentationcited by 245
  11. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  12. 2021Decoupling Zero-Shot Semantic Segmentationcited by 199
  13. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
  14. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  15. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 167
  16. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  17. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  18. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  19. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  20. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  21. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  22. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  23. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  24. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  25. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  26. 2017Open Vocabulary Scene Parsingcited by 76
  27. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  28. 2022Unsupervised Object Localization: Observing the Background to Discover Objectscited by 66
  29. 2020Toward unsupervised, multi-object discovery in large-scale image collectionscited by 65
  30. 2018Learning to Detectcited by 63
  31. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  32. 2022Unsupervised Salient Object Detection with Spectral Cluster Votingcited by 56
  33. 2021Large-Scale Unsupervised Object Discoverycited by 54
  34. 2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
  35. 2023ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistencycited by 51
  36. 2022MOVE: Unsupervised Movable Object Segmentation and Detectioncited by 20
  37. 2023Zero-guidance Segmentation Using Zero Segment Labelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 12; 2026: 11.

39 papers.

Sort:

By area:

  1. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  2. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  3. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  4. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 44
  5. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  6. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  7. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  8. 2024Open Panoramic Segmentationcited by 17
  9. 2023Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Surveycited by 16
  10. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  11. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  12. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
27 more
  1. 2024Tuning-free Universally-Supervised Semantic Segmentationcited by 6
  2. 2024Bayesian Active Learning for Semantic Segmentationcited by 5
  3. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  4. 2025LOSC: LiDAR Open-voc Segmentation Consolidatorcited by 5
  5. 2025RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Modelscited by 5
  6. 2024Composed Image Retrieval for Training-Free Domain Conversioncited by 4
  7. 2025FLOSS: Free Lunch in Open-vocabulary Semantic Segmentationcited by 4
  8. 2024A Simple Framework for Open-Vocabulary Zero-Shot Segmentationcited by 3
  9. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  10. 2024Annotation Free Semantic Segmentation with Vision Foundation Modelscited by 2
  11. 2024Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Modelscited by 2
  12. 2025NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentationcited by 2
  13. 2026IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidancecited by 2
  14. 2026Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentationcited by 2
  15. 2025FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentationcited by 1
  16. 2025Synthetic Captions for Open-Vocabulary Zero-Shot Segmentationcited by 1
  17. 2026DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentationcited by 1
  18. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Modelscited by 0
  19. 2025A Survey on Training-free Open-Vocabulary Semantic Segmentationcited by 0
  20. 2025Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrationscited by 0
  21. 2026Open Vocabulary Panoptic Segmentation With Retrieval Augmentationcited by 0
  22. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?cited by 0
  23. 2026MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generationcited by 0
  24. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0
  25. 2026PrAda: Few-Shot Visual Adaptation for Text-Prompted Segmentationcited by 0
  26. 2026Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generatorcited by 0
  27. 2026Solving the Needle-in-a-Haystack Problem in Mammography Vision-Language Model with Differentiable Subset Samplingcited by 0