mlsanityData to 7 Oct 2026

A Comprehensive Review of Knowledge Distillation in Computer Vision

arXiv 2404.00936 ↗First posted 1 Apr 2024cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Deep learning techniques have been demonstrated to surpass preceding cutting-edge machine learning techniques in recent years, with computer vision being one of the most prominent examples. However, deep learning models suffer from significant drawbacks when deployed in resource-constrained environments due to their large model size and high complexity. Knowledge Distillation is one of the prominent solutions to overcome this challenge. This review paper examines the current state of research on knowledge distillation, a technique for compressing complex models into smaller and simpler ones. The paper provides an overview of the major principles and techniques associated with knowledge distillation and reviews the applications of knowledge distillation in the domain of computer vision. The review focuses on the benefits of knowledge distillation, as well as the problems that must be overcome to improve its effectiveness.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 45 of its 78 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

45 of 78 references.

References · 45

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  9. 2019Relational Knowledge Distillationcited by 873
  10. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  11. 2020ResNeSt: Split-Attention Networkscited by 445
  12. 2022Decoupled Knowledge Distillationcited by 380
33 more
  1. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  2. 2018Model compression via distillation and quantizationcited by 287
  3. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  4. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  5. 2015Cross Modal Distillation for Supervision Transfercited by 173
  6. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
  7. 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 106
  8. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  9. 2022A very preliminary analysis of DALL-E 2cited by 82
  10. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  11. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  12. 2018Low-Resolution Face Recognitioncited by 64
  13. 2020Feature-map-level Online Adversarial Knowledge Distillationcited by 62
  14. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  15. 2018Low-resolution Face Recognition in the Wild via Selective Knowledge Distillationcited by 55
  16. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  17. 2022Cross-Architecture Knowledge Distillationcited by 40
  18. 2018Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detectioncited by 37
  19. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  20. 2019Graph-based Knowledge Distillation by Multi-head Attention Networkcited by 31
  21. 2020Lifelong Language Knowledge Distillationcited by 31
  22. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  23. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  24. 2022Self-Distillation from the Last Mini-Batch for Consistency Regularizationcited by 29
  25. 2022Switchable Online Knowledge Distillationcited by 29
  26. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  27. 2022Learning to Detect and Segment for Open Vocabulary Object Detectioncited by 22
  28. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  29. 2017DenseNet for Dense Flowcited by 15
  30. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesiscited by 14
  31. 2022FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentationcited by 14
  32. 2024Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modalitycited by 3
  33. 2024MCF-VC: Mitigate Catastrophic Forgetting in Class-Incremental Learning for Multimodal Video Captioningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  2. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  3. 2024To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillationcited by 4
  4. 2025Adaptive Temperature Based on Logits Correlation in Knowledge Distillationcited by 2
  5. 2025Temperature-Driven Robust Disease Detection in Brain and Gastrointestinal Disorders via Context-Aware Adaptive Knowledge Distillationcited by 1
  6. 2025Learning Using Privileged Information for Litter Detectioncited by 1
  7. 2026Enhancing Object Detection with Privileged Information: A Model-Agnostic Teacher-Student Approachcited by 1
  8. 2025MobiAct: Efficient MAV Action Recognition Using MobileNetV4 with Contrastive Learning and Knowledge Distillationcited by 0
  9. 2026Pareto-guided Pipeline for Distilling Featherweight AI Agents in Mobile MOBA Gamescited by 0
  10. 2026SelfDR: Self-Distillation from Reasoning for LLM-Based Recommendationcited by 0