mlsanityData to 7 Oct 2026

Does Knowledge Distillation Really Work?

arXiv 2106.05945 ↗First posted 10 Jun 2021cs.LG stat.MLcited by 187

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

35 of 56 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
23 more
  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  3. 2016Sequence-Level Knowledge Distillationcited by 761
  4. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  5. 2018Born Again Neural Networkscited by 539
  6. 2013Do Deep Nets Really Need to be Deep?cited by 466
  7. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  8. 2019On the Efficacy of Knowledge Distillationcited by 377
  9. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  10. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  11. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  12. 2019Ensemble Distribution Distillationcited by 158
  13. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  14. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  15. 2019Adversarially Robust Distillationcited by 124
  16. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  17. 2017Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillationcited by 62
  18. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  19. 2017Between-class Learning for Image Classificationcited by 50
  20. 2018Improving the Interpretability of Deep Neural Networks with Knowledge Distillationcited by 37
  21. 2018Adversarial Teacher-Student Learning for Unsupervised Domain Adaptationcited by 27
  22. 2020Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillationcited by 27
  23. 2015Distilling Knowledge from Deep Networks with Applications to Healthcare Domaincited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 33; 2023: 43; 2024: 41; 2025: 32; 2026: 30.

187 papers.

Sort:

By area:

  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2022FlexiViT: One Model for All Patch Sizescited by 151
  4. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  5. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  6. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  7. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  8. 2022Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillationcited by 61
  9. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 61
  10. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  11. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  12. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
38 more
  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  2. 2021A Survey on Green Deep Learningcited by 52
  3. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  4. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  5. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  6. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  7. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  8. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  9. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  10. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
  11. 2022A Survey on Cross-Lingual Summarizationcited by 38
  12. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  13. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  14. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  15. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  16. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 26
  17. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  18. 2021Watermarking Graph Neural Networks based on Backdoor Attackscited by 25
  19. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  20. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  21. 2022Revisiting Self-Distillationcited by 23
  22. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  23. 2023GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Mapscited by 23
  24. 2022Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Surveycited by 22
  25. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  26. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  27. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  28. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  29. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  30. 2023ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Modelcited by 16
  31. 2024Training LLMs over Neurally Compressed Textcited by 16
  32. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  33. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  34. 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillationcited by 14
  35. 2024Good Teachers Explain: Explanation-Enhanced Knowledge Distillationcited by 14
  36. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  37. 2022Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticketcited by 13
  38. 2024How to Train the Teacher Model for Effective Knowledge Distillationcited by 13

50 most cited of 187 All 187 (JSON)