mlsanityData to 7 Oct 2026

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

arXiv 2208.12262 ↗First posted 25 Aug 2022cs.CVcited by 187

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a simple yet effective framework MaskCLIP, which incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. The core idea of masked self-distillation is to distill representation from a full image to the representation predicted from a masked image. Such incorporation enjoys two vital benefits. First, masked self-distillation targets local patch representation learning, which is complementary to vision-language contrastive focusing on text-related representation. Second, masked self-distillation is also consistent with vision-language contrastive from the perspective of training objective as both utilize the visual encoder for feature aligning, and thus is able to learn local semantics getting indirect supervision from the language. We provide specially designed experiments with a comprehensive analysis to validate the two benefits. Symmetrically, we also introduce the local semantic supervision into the text branch, which further improves the pretraining performance. With extensive experiments, we show that MaskCLIP, when applied to various challenging downstream tasks, achieves superior results in linear probing, finetuning, and zero-shot performance with the guidance of the language encoder. Code will be release at urlhttps://github.com/LightDXY/MaskCLIP.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 62 of its 83 references that are in the map. First posted on arXiv 25 Aug 2022.

Builds on

62 of 83 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  19. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  20. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  21. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  22. 2017Large Batch Training of Convolutional Networkscited by 529
  23. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  24. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  25. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  26. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  27. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  28. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  29. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  30. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  31. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  32. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  33. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  34. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  35. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  36. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  37. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  38. 2021BEVT: BERT Pretraining of Video Transformerscited by 169

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 50; 2024: 48; 2025: 52; 2026: 29.

187 papers.

Sort:

By area:

  1. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  2. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  3. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 128
  4. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  5. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  6. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  7. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  8. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  9. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  10. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  11. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  12. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
38 more
  1. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  2. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  3. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  4. 2023ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistencycited by 50
  5. 2022X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusioncited by 49
  6. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  7. 2024GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understandingcited by 46
  8. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  9. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  10. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  11. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  12. 2022CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNetcited by 35
  13. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentationcited by 34
  14. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  15. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  16. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 33
  17. 2022Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Informationcited by 31
  18. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  19. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  20. 2023Segment and Caption Anythingcited by 30
  21. 2024GalLoP: Learning Global and Local Prompts for Vision-Language Modelscited by 29
  22. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  23. 2024Masked Image Modeling: A Surveycited by 28
  24. 2023CapDet: Unifying Dense Captioning and Open-World Detection Pretrainingcited by 26
  25. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  26. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  27. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Modelscited by 24
  28. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  29. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  30. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformerscited by 24
  31. 2022Attentive Mask CLIPcited by 23
  32. 2023Supervised Masked Knowledge Distillation for Few-Shot Transformerscited by 23
  33. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 23
  34. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  35. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  36. 2023Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classificationcited by 21
  37. 2023Improved baselines for vision-language pre-trainingcited by 20
  38. 2023MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentationcited by 18

50 most cited of 187 All 187 (JSON)