mlsanityData to 7 Oct 2026

Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

arXiv 2210.04150 ↗First posted 9 Oct 2022cs.CV cs.LGcited by 560

Not on the map. Why?

Cited by 108 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We identify the performance bottleneck of this paradigm to be the pre-trained CLIP model, since it does not perform well on masked images. To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. We collect training data by mining an existing image-caption dataset (e.g., COCO Captions), using CLIP to match masked image regions to nouns in the image captions. Compared with the more precise and manually annotated segmentation labels with fixed classes (e.g., COCO-Stuff), we find our noisy but diverse dataset can better retain CLIP's generalization ability. Along with finetuning the entire model, we utilize the "blank" areas in masked images using a method we dub mask prompt tuning. Experiments demonstrate mask prompt tuning brings significant improvement without modifying any weights of CLIP, and it can further improve a fully finetuned model. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-of-the-art. For the first time, open-vocabulary generalist models match the performance of supervised specialist models in 2017 without dataset-specific adaptations.

Key facts. Cited by 560 papers in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

34 of 51 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
22 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2022Language-driven Semantic Segmentationcited by 711
  11. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  12. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  13. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  14. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  15. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  16. 2020Rethinking Pre-training and Self-trainingcited by 352
  17. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  18. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  19. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  20. 2019Zero-Shot Semantic Segmentationcited by 245
  21. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  22. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 112; 2024: 184; 2025: 149; 2026: 112.

560 papers.

Sort:

By area:

  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  5. 2023LERF: Language Embedded Radiance Fieldscited by 538
  6. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  7. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  8. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  9. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  10. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  11. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  12. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
38 more
  1. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  2. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  3. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  4. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  5. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  6. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  7. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  8. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  9. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  10. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  11. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  12. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  13. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  14. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  15. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  16. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  17. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  18. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  19. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  20. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  21. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  22. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  23. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  24. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  25. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  26. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  27. 2023Fine-Grained Visual Promptingcited by 81
  28. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  29. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  30. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  31. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  32. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  33. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  34. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  35. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  36. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  37. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  38. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68

50 most cited of 560 All 560 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model
  2. 2025From Open-Vocabulary to Vocabulary-Free Semantic Segmentation
  3. 2023Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations
  4. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentation
  5. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
  6. 2025Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentation
  7. 2024MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment
  8. 2025Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation