mlsanityData to 7 Oct 2026

CLIP-KD: An Empirical Study of CLIP Model Distillation

arXiv 2307.12732 ↗First posted 24 Jul 2023cs.CVcited by 87

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has become a promising language-supervised visual pre-training framework. This paper aims to distill small CLIP models supervised by a large teacher CLIP model. We propose several distillation strategies, including relation, feature, gradient and contrastive paradigms, to examine the effectiveness of CLIP-Knowledge Distillation (KD). We show that a simple feature mimicry with Mean Squared Error loss works surprisingly well. Moreover, interactive contrastive learning across teacher and student encoders is also effective in performance improvement. We explain that the success of CLIP-KD can be attributed to maximizing the feature similarity between teacher and student. The unified method is applied to distill several student models trained on CC3M+12M. CLIP-KD improves student CLIP models consistently over zero-shot ImageNet classification and cross-modal retrieval benchmarks. When using ViT-L/14 pretrained on Laion-400M as the teacher, CLIP-KD achieves 57.5% and 55.4% zero-shot top-1 ImageNet accuracy over ViT-B/16 and ResNet-50, surpassing the original CLIP without KD by 20.5% and 20.1% margins, respectively. Our code is released on https://github.com/winycg/CLIP-KD.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 51 of its 63 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

51 of 63 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  13. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  14. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  15. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  16. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  17. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  18. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  19. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  20. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  21. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  22. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  23. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  24. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  25. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  26. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  27. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  28. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  29. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  30. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  31. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  32. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  33. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  34. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  35. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  36. 2021G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitationcited by 27
  37. 2022Attentive Mask CLIPcited by 23
  38. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 39; 2026: 34.

87 papers.

Sort:

By area:

  1. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 53
  2. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  3. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  4. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 21
  5. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  6. 2024MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Modelscited by 20
  7. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  8. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  9. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  10. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Modelscited by 11
  11. 2026Distilling Time Series Foundation Models for Efficient Forecastingcited by 11
  12. 2025Neodragon: Mobile Video Generation using Diffusion Transformercited by 10
38 more
  1. 2025SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillationcited by 9
  2. 2025Enhancing Image Generation Fidelity via Progressive Promptscited by 7
  3. 2025DCA: Dividing and Conquering Amnesia in Incremental Object Detectioncited by 7
  4. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  5. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  6. 2025Multi-party Collaborative Attention Control for Image Customizationcited by 6
  7. 2025Single Domain Generalization for Few-Shot Counting via Universal Representation Matchingcited by 6
  8. 2025Finger Pose Estimation for Under-screen Fingerprint Sensorcited by 5
  9. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
  10. 2024Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Datacited by 4
  11. 2025TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledgecited by 4
  12. 2025Multimodal Medical Image Binding via Shared Text Embeddingscited by 3
  13. 2025FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolutioncited by 3
  14. 2024Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learningcited by 2
  15. 2024IOR: Inversed Objects Replay for Incremental Object Detectioncited by 2
  16. 2025A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Featurescited by 2
  17. 2025Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Studentcited by 2
  18. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrievalcited by 2
  19. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiencycited by 2
  20. 2026RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioningcited by 2
  21. 2024CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teacherscited by 1
  22. 2024Prototype-Driven Multi-Feature Generation for Visible-Infrared Person Re-identificationcited by 1
  23. 2024ECG-guided individual identification via PPGcited by 1
  24. 2025Efficient Continual Learning through Frequency Decomposition and Integrationcited by 1
  25. 2025Crossmodal Knowledge Distillation with WordNet-Relaxed Text Embeddings for Robust Image Classificationcited by 1
  26. 2025AGATE: Stealthy Black-box Watermarking for Multimodal Model Copyright Protectioncited by 1
  27. 2025A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuningcited by 1
  28. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  29. 2025FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptationcited by 1
  30. 2025Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Studycited by 1
  31. 2025ProCLIP: Progressive Vision-Language Alignment via LLM-based Embeddercited by 1
  32. 2025A Systematic Study of Model Extraction Attacks on Graph Foundation Modelscited by 1
  33. 2025From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformerscited by 1
  34. 2025Foundry: Distilling 3D Foundation Models for the Edgecited by 1
  35. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Modelscited by 1
  36. 2026Robustness of Vision Language Models Against Split-Image Harmful Input Attackscited by 1
  37. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  38. 2024Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computerscited by 0

50 most cited of 87 All 87 (JSON)