mlsanityData to 7 Oct 2026

Renovating Names in Open-Vocabulary Segmentation Benchmarks

arXiv 2403.09593 ↗First posted 14 Mar 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Names are essential to both human cognition and vision-language models. Open-vocabulary models utilize class names as text prompts to generalize to categories unseen during training. However, the precision of these names is often overlooked in existing datasets. In this paper, we address this underexplored problem by presenting a framework for "renovating" names in open-vocabulary segmentation benchmarks (RENOVATE). Our framework features a renaming model that enhances the quality of names for each visual segment. Through experiments, we demonstrate that our renovated names help train stronger open-vocabulary models with up to 15% relative improvement and significantly enhance training efficiency with improved data quality. We also show that our renovated names improve evaluation by better measuring misclassification and enabling fine-grained model analysis. We will provide our code and relabelings for several popular segmentation datasets (MS COCO, ADE20K, Cityscapes) to the research community.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 71 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

47 of 71 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Vocabulary-free Image Classification1 Jun
  2. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  3. 2022Visual Classification via Description from Large Language Models13 Oct
  4. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
31 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  11. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  12. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  13. 2016FastText.zip: Compressing text classification modelscited by 331
  14. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  15. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  16. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  17. 2019Zero-Shot Semantic Segmentationcited by 245
  18. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  19. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  20. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  21. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  22. 2019Text Classification Algorithms: A Surveycited by 163
  23. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 109
  24. 2017Open Vocabulary Scene Parsingcited by 76
  25. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  26. 2022Improving Zero-shot Generalization and Robustness of Multi-modal Modelscited by 44
  27. 2023AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregationcited by 29
  28. 2023Multi-Modal Prototypes for Open-World Semantic Segmentationcited by 22
  29. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 14
  30. 2023Learning to Name Classes for Vision and Language Modelscited by 12
  31. 2023Rethinking Evaluation Metrics of Open-Vocabulary Segmentaioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Modelscited by 23
  2. 2025FLOSS: Free Lunch in Open-vocabulary Semantic Segmentationcited by 4
  3. 2025Image Recognition with Vision and Language Embeddings of VLMscited by 2
  4. 2025Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentationcited by 1
  5. 2025Scalable Object Detection in the Car Interior With Vision Foundation Modelscited by 0
  6. 2026DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Modelscited by 0
  7. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Transferable and Principled Efficiency for Open-Vocabulary Segmentation
  2. 2025From Open-Vocabulary to Vocabulary-Free Semantic Segmentation
  3. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation
  4. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentation
  5. 2025Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation
  6. 2023Diffusion Models for Open-Vocabulary Segmentation
  7. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
  8. 2024OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation