mlsanityData to 7 Oct 2026

Auto-Vocabulary Semantic Segmentation

arXiv 2312.04539 ↗First posted 7 Dec 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Open-Vocabulary Segmentation (OVS) methods are capable of performing semantic segmentation without relying on a fixed vocabulary, and in some cases, without training or fine-tuning. However, OVS methods typically require a human in the loop to specify the vocabulary based on the task or dataset at hand. In this paper, we introduce Auto-Vocabulary Semantic Segmentation (AVS), advancing open-ended image understanding by eliminating the necessity to predefine object categories for segmentation. Our approach, AutoSeg, presents a framework that autonomously identifies relevant class names using semantically enhanced BLIP embeddings and segments them afterwards. Given that open-ended object category predictions cannot be directly compared with a fixed ground truth, we develop a Large Language Model-based Auto-Vocabulary Evaluator (LAVE) to efficiently evaluate the automatically generated classes and their corresponding segments. With AVS, our method sets new benchmarks on datasets PASCAL VOC, Context, ADE20K, and Cityscapes, while showing competitive performance to OVS methods that require specified class names.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  11. 2022Language-driven Semantic Segmentationcited by 711
  12. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
24 more
  1. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  3. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  4. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  5. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  6. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  7. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  8. 2019Zero-Shot Semantic Segmentationcited by 245
  9. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  10. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  11. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  12. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  13. 2019Detecting the Unexpected via Image Resynthesiscited by 137
  14. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  15. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  16. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  17. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  18. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  19. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Modelscited by 39
  20. 2023Open-vocabulary Panoptic Segmentation with Embedding Modulationcited by 39
  21. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  22. 2023Zero-guidance Segmentation Using Zero Segment Labelscited by 18
  23. 2022What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputscited by 11
  24. 2023A Language-Guided Benchmark for Weakly Supervised Open Vocabulary Semantic Segmentationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 4.

8 papers.

  1. 2024TAG: Guidance-free Open-Vocabulary Semantic Segmentationcited by 6
  2. 20243D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentationcited by 5
  3. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentationcited by 1
  4. 2025Language-Guided Open-World Anomaly Segmentationcited by 0
  5. 2025Auto-Vocabulary 3D Object Detectioncited by 0
  6. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0
  7. 2026WOW-Seg: A Word-free Open World Segmentation Modelcited by 0
  8. 2026LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentationcited by 0