mlsanityData to 7 Oct 2026

Supervision Complexity and its Role in Knowledge Distillation

arXiv 2301.12245 ↗First posted 28 Jan 2023cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps. In order to study the generalization behavior of a distilled student, we propose a new theoretical framework that leverages supervision complexity: a measure of alignment between teacher-provided supervision and the student's neural tangent kernel. The framework highlights a delicate interplay among the teacher's accuracy, the student's margin with respect to the teacher predictions, and the complexity of the teacher predictions. Specifically, it provides a rigorous justification for the utility of various techniques that are prevalent in the context of distillation, such as early stopping and temperature scaling. Our analysis further suggests the use of online distillation, where a student receives increasingly more complex supervision from teachers in different stages of their training. We demonstrate efficacy of online distillation and validate the theoretical findings on a range of image classification benchmarks and model architectures.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

44 of 62 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2019Searching for MobileNetV3cited by 1,984
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  9. 2019Relational Knowledge Distillationcited by 873
  10. 2019Contrastive Representation Distillationcited by 734
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  12. 2017Deep Mutual Learningcited by 621
32 more
  1. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  2. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  3. 2018Born Again Neural Networkscited by 539
  4. 2019Similarity-Preserving Knowledge Distillationcited by 507
  5. 2019On the Efficacy of Knowledge Distillationcited by 377
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  8. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  9. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  10. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  11. 2021Does Knowledge Distillation Really Work?cited by 187
  12. 2018To understand deep learning we need to understand kernel learningcited by 186
  13. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  14. 2021Towards Understanding Knowledge Distillationcited by 158
  15. 2018Large scale distributed neural network training through online distillationcited by 144
  16. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  17. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  18. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  19. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  20. 2020Understanding and Improving Knowledge Distillationcited by 83
  21. 2021Input Similarity from the Neural Network Perspectivecited by 78
  22. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  23. 2021What can linearized neural networks actually say about generalization?cited by 49
  24. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  25. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  26. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  27. 2017Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Netcited by 27
  28. 2020Subclass Distillationcited by 23
  29. 2022Better Supervisory Signals by Observing Learning Pathscited by 21
  30. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  31. 2021Follow Your Path: a Progressive Method for Knowledge Distillationcited by 9
  32. 2022Learning Representation from Neural Fisher Kernel with Low-rank Approximationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 3; 2026: 7.

19 papers.

Sort:

  1. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 21
  2. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  3. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  4. 2024The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Informationcited by 8
  5. 2023Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clusteringcited by 7
  6. 2024Learning Neural Networks with Sparse Activationscited by 6
  7. 2023Data Upcycling Knowledge Distillation for Image Super-Resolutioncited by 4
  8. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  9. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 3
  10. 2025Efficient Knowledge Distillation via Curriculum Extractioncited by 2
  11. 2025Neural Tangent Knowledge Distillation for Optical Convolutional Networkscited by 2
  12. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Modelscited by 1
7 more
  1. 2024Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Trainingcited by 0
  2. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  3. 2026Following the Teacher's Footsteps: Scheduled Checkpoint Distillation for Domain-Specific LLMscited by 0
  4. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  5. 2026Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Studycited by 0
  6. 2026JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantizationcited by 0
  7. 2026Learning Beyond Full Imitation: Task-Preserving Knowledge Distillationcited by 0