mlsanityData to 14 Oct 2026

Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models

arXiv 2311.17095 ↗First posted 28 Nov 2023cs.CV cs.AIcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

From image-text pairs, large-scale vision-language models (VLMs) learn to implicitly associate image regions with words, which prove effective for tasks like visual question answering. However, leveraging the learned association for open-vocabulary semantic segmentation remains a challenge. In this paper, we propose a simple, yet extremely effective, training-free technique, Plug-and-Play Open-Vocabulary Semantic Segmentation (PnP-OVSS) for this task. PnP-OVSS leverages a VLM with direct text-to-image cross-attention and an image-text matching loss. To balance between over-segmentation and under-segmentation, we introduce Salience Dropout; by iteratively dropping patches that the model is most attentive to, we are able to better resolve the entire extent of the segmentation mask. PnP-OVSS does not require any neural network training and performs hyperparameter tuning without the need for any segmentation annotations, even for a validation set. PnP-OVSS demonstrates substantial improvements over comparable baselines (+26.2% mIoU on Pascal VOC, +20.5% mIoU on MS COCO, +3.1% mIoU on COCO Stuff and +3.0% mIoU on ADE20K). Our codebase is at https://github.com/letitiabanana/PnP-OVSS.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 90 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

70 of 90 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation16 Mar
  2. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  3. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  12. 2023Vision Transformers Need Registerscited by 871
38 more
  1. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  3. 2021Extract Free Dense Labels from CLIPcited by 583
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  5. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  6. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  8. 2019Zero-Shot Semantic Segmentationcited by 245
  9. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  10. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  11. 2017Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localizationcited by 207
  12. 2017Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approachcited by 201
  13. 2016Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentationcited by 198
  14. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  15. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  16. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  17. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
  18. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  19. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  20. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 167
  21. 2018Adversarial Complementary Learning for Weakly Supervised Object Localizationcited by 166
  22. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  23. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  24. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  25. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  26. 2018On Regularized Losses for Weakly-supervised CNN Segmentationcited by 121
  27. 2022Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformerscited by 118
  28. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  29. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  30. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  31. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  32. 2018Self-Erasing Network for Integral Object Attentioncited by 95
  33. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  34. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  35. 2019Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approachcited by 88
  36. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  37. 2022Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentationcited by 87
  38. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 6.

15 papers.

Sort:

By area:

  1. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  2. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 24
  3. 2025ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentationcited by 9
  4. 2024Enhancing Vision-Language Tracking by Effectively Converting Textual Cues into Visual Cuescited by 8
  5. 2025PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?cited by 4
  6. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMscited by 4
  7. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  8. 2025NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentationcited by 2
  9. 2026DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentationcited by 2
  10. 2025Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentationcited by 1
  11. 2026DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentationcited by 1
  12. 2025MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinkingcited by 0
3 more
  1. 2026Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentationcited by 0
  2. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0
  3. 2026MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentation
  2. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
  3. 2024Transferable and Principled Efficiency for Open-Vocabulary Segmentation
  4. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  5. 2026dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
  6. 2024OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation
  7. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation
  8. 2023Towards Open Vocabulary Learning: A Survey