mlsanityData to 14 Oct 2026

What Holds Back Open-Vocabulary Segmentation?

arXiv 2508.04211 ↗First posted 6 Aug 2025cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Standard segmentation setups are unable to deliver models that can recognize concepts outside the training taxonomy. Open-vocabulary approaches promise to close this gap through language-image pretraining on billions of image-caption pairs. Unfortunately, we observe that the promise is not delivered due to several bottlenecks that have caused the performance to plateau for almost two years. This paper proposes novel oracle components that identify and decouple these bottlenecks by taking advantage of the groundtruth information. The presented validation experiments deliver important empirical findings that provide a deeper insight into the failures of open-vocabulary models and suggest prominent approaches to unlock the future research.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 58 of its 64 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

58 of 64 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2017Mask R-CNNcited by 7,277
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
38 more
  1. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2015Fast R-CNNcited by 2,933
  4. 2016Pyramid Scene Parsing Networkcited by 2,479
  5. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  6. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  10. 2018Panoptic Segmentationcited by 802
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  13. 2021Extract Free Dense Labels from CLIPcited by 583
  14. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  15. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  16. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  17. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  18. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  19. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  20. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  21. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  22. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  23. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  24. 2018DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Imagescited by 277
  25. 2023Demystifying CLIP Datacited by 270
  26. 2023Data Filtering Networkscited by 250
  27. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  28. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  29. 2021Decoupling Zero-Shot Semantic Segmentationcited by 199
  30. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  31. 2014Simultaneous Detection and Segmentationcited by 159
  32. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  33. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  34. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  35. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  36. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  37. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  38. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 97

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentationcited by 2
  2. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?cited by 0
  3. 2026SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentationcited by 0
  4. 2026TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perceptioncited by 0