mlsanityData to 14 Oct 2026

Extract Free Dense Labels from CLIP

arXiv 2112.01071 ↗First posted 2 Dec 2021cs.CV cs.CLcited by 583

Not on the map. Why?

Cited by 142 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition. Many recent studies leverage the pre-trained CLIP models for image-level classification and manipulation. In this paper, we wish examine the intrinsic potential of CLIP for pixel-level dense prediction, specifically in semantic segmentation. To this end, with minimal modification, we show that MaskCLIP yields compelling segmentation results on open concepts across various datasets in the absence of annotations and fine-tuning. By adding pseudo labeling and self-training, MaskCLIP+ surpasses SOTA transductive zero-shot semantic segmentation methods by large margins, e.g., mIoUs of unseen classes on PASCAL VOC/PASCAL Context/COCO Stuff are improved from 35.6/20.7/30.3 to 86.1/66.7/54.7. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a new reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation. Source code is available at https://github.com/chongzhou96/MaskCLIP.

Key facts. Cited by 583 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2016Pyramid Scene Parsing Networkcited by 2,479
29 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  4. 2014How transferable are features in deep neural networks?cited by 1,225
  5. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  9. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  10. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  11. 2020Semi-Supervised Semantic Segmentation with Cross-Consistency Trainingcited by 376
  12. 2020Meta Pseudo Labelscited by 374
  13. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  14. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  15. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  16. 2019Zero-Shot Semantic Segmentationcited by 245
  17. 2020PseudoSeg: Designing Pseudo Labels for Semantic Segmentationcited by 180
  18. 2018Adversarial Learning for Semi-Supervised Semantic Segmentationcited by 173
  19. 2019Semi-Supervised Semantic Segmentation with High- and Low-level Consistencycited by 147
  20. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  21. 2020Learning Visual Representations with Caption Annotationscited by 123
  22. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  23. 2019Learning to Self-Train for Semi-Supervised Few-Shot Classificationcited by 83
  24. 2017Open Vocabulary Scene Parsingcited by 76
  25. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  26. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43
  27. 2020Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentationcited by 38
  28. 2018Semi-Supervised Semantic Image Segmentation with Self-correcting Networkscited by 37
  29. 2021Inverse Problems Leveraging Pre-trained Contrastive Representationscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 132; 2024: 170; 2025: 159; 2026: 108.

583 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  4. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
  5. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  6. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  7. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  8. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  9. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  10. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  11. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  12. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
38 more
  1. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  2. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  3. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  4. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  5. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  6. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 143
  7. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  8. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  9. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  10. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  11. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  12. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  13. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  14. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  15. 2023Iterative Prompt Learning for Unsupervised Backlit Image Enhancementcited by 107
  16. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  17. 2024PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detectioncited by 98
  18. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 97
  19. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 96
  20. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  21. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  22. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  23. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  24. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 91
  25. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  26. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  27. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  28. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  29. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  30. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  31. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  32. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  33. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  34. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  35. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  36. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  37. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  38. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61

50 most cited of 583 All 583 (JSON)