mlsanityData to 7 Oct 2026

ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

arXiv 2408.04145 ↗First posted 8 Aug 2024cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) models excel in integrating semantic information between images and text through contrastive learning techniques. It has achieved remarkable performance in various multimodal tasks. However, the deployment of large CLIP models is hindered in resource-limited environments, while smaller models frequently fail to meet the performance benchmarks required for practical applications. In this paper, we propose a novel approach, ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model, which aims to comprehensively distill the knowledge from a large teacher CLIP model into a smaller student model, ensuring comparable performance with significantly reduced parameters. ComKD-CLIP is composed of two key mechanisms: Image Feature Alignment (IFAlign) and Educational Attention (EduAttention). IFAlign makes the image features extracted by the student model closely match those extracted by the teacher model, enabling the student to learn teacher's knowledge of extracting image features. EduAttention explores the cross-relationships between text features extracted by the teacher model and image features extracted by the student model, enabling the student model to learn how the teacher model integrates text-image features. In addition, ComKD-CLIP can refine the knowledge distilled from IFAlign and EduAttention by leveraging the text-image feature fusion results of the teacher model, ensuring the student model accurately absorbs the teacher's knowledge. Extensive experiments conducted on 11 datasets have demonstrated the superiority of the proposed method.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

23 of 39 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  10. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  11. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  12. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
11 more
  1. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  2. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  3. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  4. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  5. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  6. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  7. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  8. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  9. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  10. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learningcited by 15
  11. 2023Improving CLIP Robustness with Knowledge Distillation and Self-Trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  2. 2025Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Modelscited by 6
  3. 2025TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledgecited by 4
  4. 2025DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detectioncited by 4
  5. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  6. 2024CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidancecited by 0
  7. 2025Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noisecited by 0
  8. 2026TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddingscited by 0
  9. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representationscited by 0