mlsanityData to 14 Oct 2026

Open-Vocabulary Universal Image Segmentation with MaskCLIP

arXiv 2208.08984 ↗First posted 18 Aug 2022cs.CVcited by 132

Not on the map. Why?

Cited by 36 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we tackle an emerging computer vision task, open-vocabulary universal image segmentation, that aims to perform semantic/instance/panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time. We first build a baseline method by directly adopting pre-trained CLIP models without finetuning or distillation. We then develop MaskCLIP, a Transformer-based approach with a MaskCLIP Visual Encoder, which is an encoder-only module that seamlessly integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction. MaskCLIP learns to efficiently and effectively utilize pre-trained partial/dense CLIP features within the MaskCLIP Visual Encoder that avoids the time-consuming student-teacher training process. MaskCLIP outperforms previous methods for semantic/instance/panoptic segmentation on ADE20K and PASCAL datasets. We show qualitative illustrations for MaskCLIP with online custom categories. Project website: https://maskclip.github.io.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). Builds on 37 of its 43 references that are in the map. First posted on arXiv 18 Aug 2022.

Builds on

37 of 43 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2017Mask R-CNNcited by 7,277
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  8. 2021Grounded Language-Image Pre-trainingcited by 1,156
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
25 more
  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  2. 2018Panoptic Segmentationcited by 802
  3. 2022Language-driven Semantic Segmentationcited by 717
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  5. 2019Panoptic Feature Pyramid Networkscited by 645
  6. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  7. 2021Extract Free Dense Labels from CLIPcited by 583
  8. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  11. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  12. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 316
  13. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 306
  14. 2014Towards Open World Recognitioncited by 226
  15. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  16. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  17. 2021Decoupling Zero-Shot Semantic Segmentationcited by 199
  18. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  19. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  20. 2021Open-World Entity Segmentationcited by 74
  21. 2018Attention-guided Unified Network for Panoptic Segmentationcited by 71
  22. 2022X-DETR: A Versatile Architecture for Instance-wise Vision-Language Taskscited by 43
  23. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  24. 2019Learning Instance Occlusion for Panoptic Segmentationcited by 30
  25. 2017Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 52; 2025: 36; 2026: 27.

132 papers.

Sort:

By area:

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  3. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  4. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  5. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  6. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  8. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  9. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  10. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 95
  11. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  12. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
38 more
  1. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  2. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 61
  3. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  4. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  5. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  6. 2024Open-Vocabulary Semantic Segmentation with Image Embedding Balancingcited by 35
  7. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  8. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  9. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  10. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 29
  11. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 25
  12. 2024SegLLM: Multi-round Reasoning Segmentationcited by 25
  13. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 20
  14. 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 20
  15. 2023Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Modelscited by 19
  16. 2023Few-Shot Panoptic Segmentation With Foundation Modelscited by 18
  17. 2024Open Panoramic Segmentationcited by 17
  18. 2024Effective SAM Combination for Open-Vocabulary Semantic Segmentationcited by 17
  19. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  20. 2023UGainS: Uncertainty Guided Anomaly Instance Segmentationcited by 15
  21. 2024USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentationcited by 14
  22. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  23. 20243D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillationcited by 13
  24. 2024UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understandingcited by 13
  25. 2024CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentationcited by 13
  26. 2024CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenescited by 13
  27. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  28. 2024MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentationcited by 11
  29. 2025Laser: Efficient Language-Guided Segmentation in Neural Radiance Fieldscited by 10
  30. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  31. 2024PosSAM: Panoptic Open-vocabulary Segment Anythingcited by 9
  32. 2024MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignmentcited by 9
  33. 2024Generalization Boosted Adapter for Open-Vocabulary Segmentationcited by 9
  34. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  35. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  36. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 9
  37. 2025DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentationcited by 8
  38. 2025SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentationcited by 8

50 most cited of 132 All 132 (JSON)