mlsanityData to 14 Oct 2026

VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition

arXiv 2408.16930 ↗First posted 29 Aug 2024cs.CVcited by 7

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

For visual recognition, knowledge distillation typically involves transferring knowledge from a large, well-trained teacher model to a smaller student model. In this paper, we introduce an effective method to distill knowledge from an off-the-shelf vision-language model (VLM), demonstrating that it provides novel supervision in addition to those from a conventional vision-only teacher model. Our key technical contribution is the development of a framework that generates novel text supervision and distills free-form text into a vision encoder. We showcase the effectiveness of our approach, termed VLM-KD, across various benchmark datasets, showing that it surpasses several state-of-the-art long-tail visual classifiers. To our knowledge, this work is the first to utilize knowledge distillation with text supervision generated by an off-the-shelf VLM and apply it to vanilla randomly initialized vision encoders.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 69 of its 86 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

69 of 86 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2019Contrastive Multiview Codingcited by 1,301
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  4. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  6. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  7. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  8. 2020Long-tail learning via logit adjustmentcited by 618
  9. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 400
  10. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  11. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  12. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  13. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  14. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  15. 2020Long-tailed Recognition by Routing Diverse Distribution-Aware Expertscited by 271
  16. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  17. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  18. 2021Parametric Contrastive Learningcited by 217
  19. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  20. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  21. 2020Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classificationcited by 175
  22. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  23. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 144
  24. 2020Feature Space Augmentation for Long-Tailed Datacited by 136
  25. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  26. 2021Contrastive Learning based Hybrid Networks for Long-Tailed Image Classificationcited by 130
  27. 2021MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognitioncited by 99
  28. 2021ResLT: Residual Learning for Long-tailed Recognitioncited by 96
  29. 2021Distributional Robustness Loss for Long-tail Learningcited by 84
  30. 2021FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentationcited by 76
  31. 2023Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustmentcited by 75
  32. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  33. 2020Regularizing Deep Networks with Semantic Data Augmentationcited by 73
  34. 2021Distilling Virtual Examples for Long-tailed Recognitioncited by 72
  35. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  36. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 66
  37. 2021VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognitioncited by 65
  38. 2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 23
  2. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  3. 2025HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMscited by 3
  4. 2025Towards Long-window Anchoring in Vision-Language Model Distillationcited by 2
  5. 2025Multimodal Distillation-Driven Ensemble Learning for Long-Tailed Histopathology Whole Slide Images Analysiscited by 1
  6. 2026Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMscited by 0
  7. 2026LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders
  2. 2021Compressing Visual-linguistic Model via Knowledge Distillation
  3. 2026DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer
  4. 2023Vision-Language Models for Vision Tasks: A Survey
  5. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
  6. 2026DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
  7. 2026Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
  8. 2025Robust MLLM Unlearning via Visual Knowledge Distillation