mlsanityData to 7 Oct 2026

Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision

arXiv 2301.09121 ↗First posted 22 Jan 2023cs.CVcited by 99

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we consider the problem of open-vocabulary semantic segmentation (OVS), which aims to segment objects of arbitrary classes instead of pre-defined, closed-set categories. The main contributions are as follows: First, we propose a transformer-based model for OVS, termed as OVSegmentor, which only exploits web-crawled image-text pairs for pre-training without using any mask annotations. OVSegmentor assembles the image pixels into a set of learnable group tokens via a slot-attention based binding module, and aligns the group tokens to the corresponding caption embedding. Second, we propose two proxy tasks for training, namely masked entity completion and cross-image mask consistency. The former aims to infer all masked entities in the caption given the group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities. The latter enforces consistent mask predictions between images that contain shared entities, which encourages the model to learn visual invariance. Third, we construct CC4M dataset for pre-training by filtering CC12M with frequently appeared entities, which significantly improves training efficiency. Fourth, we perform zero-shot transfer on three benchmark datasets, PASCAL VOC 2012, PASCAL Context, and COCO Object. Our model achieves superior segmentation results over the state-of-the-art method by using only 3% data (4M vs 134M) for pre-training. Code and pre-trained models will be released for future research.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 22 Jan 2023.

Builds on

48 of 60 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
36 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2021Grounded Language-Image Pre-trainingcited by 1,147
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  7. 2020Object-Centric Learning with Slot Attentioncited by 865
  8. 2022Language-driven Semantic Segmentationcited by 711
  9. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  10. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  11. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  12. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  13. 2020Semi-Supervised Semantic Segmentation with Cross-Consistency Trainingcited by 375
  14. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  15. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  16. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  17. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  18. 2019Zero-Shot Semantic Segmentationcited by 245
  19. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  20. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  21. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  22. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  23. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  24. 2022Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentationcited by 87
  25. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  26. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  27. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  28. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  29. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  30. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  31. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41
  32. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Modelscited by 39
  33. 2022Bridging the Gap between Classification and Localization for Weakly Supervised Object Localizationcited by 25
  34. 2021Background Activation Suppression for Weakly Supervised Object Localizationcited by 22
  35. 2022CREAM: Weakly Supervised Object Localization via Class RE-Activation Mappingcited by 21
  36. 2022NamedMask: Distilling Segmenters from Complementary Foundation Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 42; 2025: 21; 2026: 12.

99 papers.

Sort:

By area:

  1. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  3. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  4. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  5. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
  6. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  7. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  8. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  9. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  10. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  11. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  12. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
38 more
  1. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  2. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  3. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  4. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  5. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 53
  6. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  7. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  8. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 43
  9. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  10. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  11. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  12. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  13. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  14. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  15. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 28
  16. 2025MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentationcited by 28
  17. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  18. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  19. 2024MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesiscited by 22
  20. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentationcited by 21
  21. 2024Image-Text Co-Decomposition for Text-Supervised Semantic Segmentationcited by 21
  22. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  23. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  24. 2024T-VSL: Text-Guided Visual Sound Source Localization in Mixturescited by 19
  25. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  26. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  27. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentationcited by 17
  28. 2024Open Panoramic Segmentationcited by 17
  29. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  30. 2025Object-Centric World Model for Language-Guided Manipulationcited by 16
  31. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  32. 2024Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentationcited by 15
  33. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  34. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  35. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  36. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  37. 20243D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidancecited by 12
  38. 2023Language-based Action Concept Spaces Improve Video Self-Supervised Learningcited by 10

50 most cited of 99 All 99 (JSON)