mlsanityData to 7 Oct 2026

Towards Understanding Knowledge Distillation

arXiv 2105.13093 ↗First posted 27 May 2021cs.LG stat.MLcited by 158

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation, i.e., one classifier being trained on the outputs of another classifier, is an empirically very successful technique for knowledge transfer between classifiers. It has even been observed that classifiers learn much faster and more reliably if trained with the outputs of another classifier as soft labels, instead of from ground truth data. So far, however, there is no satisfactory theoretical explanation of this phenomenon. In this work, we provide the first insights into the working mechanisms of distillation by studying the special case of linear and deep linear classifiers. Specifically, we prove a generalization bound that establishes fast convergence of the expected risk of a distillation-trained linear classifier. From the bound and its proof we extract three key factors that determine the success of distillation: * data geometry -- geometric properties of the data distribution, in particular class separation, has a direct influence on the convergence speed of the risk; * optimization bias -- gradient descent optimization finds a very favorable minimum of the distillation objective; and * strong monotonicity -- the expected risk of the student classifier always decreases when the size of the training set grows.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 27 May 2021.

Builds on

21 of 29 references.

References · 21

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  6. 2013Do Deep Nets Really Need to be Deep?cited by 466
  7. 2015Policy Distillationcited by 414
  8. 2018Model compression via distillation and quantizationcited by 287
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  10. 2016Deep Learning without Poor Local Minimacited by 233
  11. 2015Unifying distillation and privileged informationcited by 218
  12. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
9 more
  1. 2017Learning from Noisy Labels with Distillationcited by 160
  2. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  3. 2016Identity Matters in Deep Learningcited by 107
  4. 2017Visual Relationship Detection with Internal and External Linguistic Knowledge Distillationcited by 60
  5. 2015The Teaching Dimension of Linear Learnerscited by 19
  6. 2013Machine Teaching for Bayesian Learners in the Exponential Familycited by 18
  7. 2015Recurrent Neural Network Training with Dark Knowledge Transfercited by 8
  8. 2015Blending LSTMs into CNNscited by 5
  9. 2016Patient-Driven Privacy Control through Generalized Distillationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 28; 2023: 26; 2024: 35; 2025: 35; 2026: 20.

158 papers.

Sort:

By area:

  1. 2022Decoupled Knowledge Distillationcited by 380
  2. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  3. 2023Getting aligned on representational alignmentcited by 142
  4. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  5. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  6. 2023Better Generative Replay for Continual Federated Learningcited by 53
  7. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  8. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  9. 2022Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recoursecited by 40
  10. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  11. 2022Linkless Link Prediction via Relational Distillationcited by 39
  12. 2023Knowledge Distillation on Graphs: A Surveycited by 38
38 more
  1. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  2. 2023Progressive Visual Prompt Learning with Contrastive Feature Re-formationcited by 35
  3. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  4. 2022Distillation-Resistant Watermarking for Model Protection in NLPcited by 29
  5. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  6. 2021Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillationcited by 28
  7. 2022UniTE: Unified Translation Evaluationcited by 28
  8. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 26
  9. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  10. 2022Universal Representations: A Unified Look at Multiple Task and Domain Learningcited by 23
  11. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  12. 2022Cross-Task Knowledge Distillation in Multi-Task Recommendationcited by 22
  13. 2022Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networkscited by 21
  14. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  15. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Trainingcited by 20
  16. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 20
  17. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  18. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  19. 2021Churn Reduction via Distillationcited by 17
  20. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  21. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  22. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  23. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  24. 2022NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphscited by 14
  25. 2024Towards a theory of model distillationcited by 14
  26. 2024Understanding the Gains from Repeated Self-Distillationcited by 14
  27. 2024Threats and Defenses in Federated Learning Life Cycle: A Comprehensive Survey and Challengescited by 14
  28. 2023Binary Classification with Confidence Differencecited by 13
  29. 2024textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuningcited by 13
  30. 2024How to Train the Teacher Model for Effective Knowledge Distillationcited by 13
  31. 2022Generalized Knowledge Distillation via Relationship Matchingcited by 12
  32. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  33. 2022PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Modelscited by 11
  34. 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
  35. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  36. 2022Narrowing the Coordinate-frame Gap in Behavior Prediction Models: Distillation for Efficient and Accurate Scene-centric Motion Forecastingcited by 10
  37. 2025LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptationcited by 10
  38. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10

50 most cited of 158 All 158 (JSON)