mlsanityData to 14 Oct 2026

TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

arXiv 2309.12314 ↗First posted 21 Sep 2023cs.CVcited by 90

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. The method introduces two core techniques: affinity mimicking and weight inheritance. Affinity mimicking explores the interaction between modalities during distillation, enabling student models to mimic teachers' behavior of learning cross-modal feature alignment in a visual-linguistic affinity space. Weight inheritance transmits the pre-trained weights from the teacher models to their student counterparts to improve distillation efficiency. Moreover, we extend the method into a multi-stage progressive distillation to mitigate the loss of informative weights during extreme compression. Comprehensive experiments demonstrate the efficacy of TinyCLIP, showing that it can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. While aiming for comparable performance, distillation with weight inheritance can speed up the training by 1.4 - 7.8 × compared to training from scratch. Moreover, our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet, surpassing the original CLIP ViT-B/16 by 3.5% while utilizing only 8.9% parameters. Finally, we demonstrate the good transferability of TinyCLIP in various downstream tasks. Code and models will be open-sourced at https://aka.ms/tinyclip.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 72 of its 79 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

72 of 79 references.

References · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  14. 2021Grounded Language-Image Pre-trainingcited by 1,156
  15. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  16. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  18. 2018Universal Transformerscited by 647
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  20. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  21. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  22. 2019Deep Equilibrium Modelscited by 615
  23. 2018Rethinking the Value of Network Pruningcited by 599
  24. 2021Extract Free Dense Labels from CLIPcited by 583
  25. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  26. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  27. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  28. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  29. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  30. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  31. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  32. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  33. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  34. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  35. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  36. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 234
  37. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  38. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 30; 2025: 28; 2026: 26.

90 papers.

Sort:

By area:

  1. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  2. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  3. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  4. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  5. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  6. 2024Transfer CLIP for Generalizable Image Denoisingcited by 29
  7. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 26
  8. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  9. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discriminationcited by 21
  10. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  11. 2024Cascade Prompt Learning for Vision-Language Model Adaptationcited by 19
  12. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
38 more
  1. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Modelscited by 15
  2. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 13
  3. 2024Accessing Vision Foundation Models via ImageNet-1Kcited by 10
  4. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Modelscited by 10
  5. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 9
  6. 2024mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphscited by 9
  7. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  8. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  9. 2024Skipping Computations in Multimodal LLMscited by 7
  10. 2025CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimizationcited by 7
  11. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  12. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  13. 2024Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentationcited by 6
  14. 2024Slicing Vision Transformer for Flexible Inferencecited by 6
  15. 2025Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillationcited by 6
  16. 2023Leveraging Foundation Models to Improve Lightweight Clients in Federated Learningcited by 5
  17. 2024Simple Unsupervised Knowledge Distillation With Space Similaritycited by 5
  18. 2024Towards Real-Time Open-Vocabulary Video Instance Segmentationcited by 5
  19. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
  20. 2024Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Datacited by 4
  21. 2024Revisiting Prompt Pretraining of Vision-Language Modelscited by 4
  22. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  23. 2025TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledgecited by 4
  24. 2025SuperCLIP: CLIP with Simple Classification Supervisioncited by 4
  25. 2024LoReTrack: Efficient and Accurate Low-Resolution Transformer Trackingcited by 3
  26. 2024HyperCLIP: Adapting Vision-Language models with Hypernetworkscited by 3
  27. 2025CLIP-SENet: CLIP-based Semantic Enhancement Network for Vehicle Re-identificationcited by 3
  28. 2025Multimodal Medical Image Binding via Shared Text Embeddingscited by 3
  29. 2025WAVECLIP: Wavelet Tokenization for Adaptive-Resolution CLIPcited by 3
  30. 2026SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devicescited by 3
  31. 2023tinyCLAP: Distilling Constrastive Language-Audio Pretrained Modelscited by 2
  32. 2024SnapCap: Efficient Snapshot Compressive Video Captioningcited by 2
  33. 2024Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Modelcited by 2
  34. 2024A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scenecited by 2
  35. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognitioncited by 2
  36. 2025AGATE: Stealthy Black-box Watermarking for Multimodal Model Copyright Protectioncited by 2
  37. 2025A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Featurescited by 2
  38. 2025FrEVL: Leveraging Frozen Pretrained Embeddings for Efficient Vision-Language Understandingcited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023CLIP-KD: An Empirical Study of CLIP Model Distillation
  2. 2023DLIP: Distilling Language-Image Pre-training
  3. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
  4. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillation
  5. 2024CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teachers
  6. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger
  7. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Training
  8. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models