mlsanityData to 7 Oct 2026

Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?

arXiv 2602.23339 ↗First posted 26 Feb 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 81 of its 92 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

81 of 92 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation28 May
  2. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation5 Dec
  3. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation26 Nov
  4. 2024Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation1 Aug
  5. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference17 Jul
  6. 2024Boosting Vision-Language Models with Transduction3 Jun
  7. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models28 May
  8. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  9. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep

Other references · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
38 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  4. 2025DINOv3cited by 1,297
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  9. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  10. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  11. 2021Extract Free Dense Labels from CLIPcited by 580
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  13. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  14. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  15. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  16. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  17. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  18. 2017One-Shot Learning for Semantic Segmentationcited by 337
  19. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  20. 2023Personalize Segment Anything Model with One Shotcited by 250
  21. 2022Memorizing Transformerscited by 219
  22. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  23. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  24. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  25. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  26. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  27. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  28. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  29. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  30. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  31. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  32. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  33. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  34. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  35. 2020Generalized Few-shot Semantic Segmentationcited by 59
  36. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  37. 2021A Large-Scale Benchmark for Food Image Segmentationcited by 45
  38. 2023Towards In-context Scene Understandingcited by 45

50 most cited of 71 All 71 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  2. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  3. 2024Transferable and Principled Efficiency for Open-Vocabulary Segmentation
  4. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
  5. 2023A Language-Guided Benchmark for Weakly Supervised Open Vocabulary Semantic Segmentation
  6. 2026dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
  7. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation
  8. 2023Towards Open Vocabulary Learning: A Survey