mlsanityData to 7 Oct 2026

How to Train the Teacher Model for Effective Knowledge Distillation

arXiv 2407.18041 ↗First posted 25 Jul 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recently, it was shown that the role of the teacher in knowledge distillation (KD) is to provide the student with an estimate of the true Bayes conditional probability density (BCPD). Notably, the new findings propose that the student's error rate can be upper-bounded by the mean squared error (MSE) between the teacher's output and BCPD. Consequently, to enhance KD efficacy, the teacher should be trained such that its output is close to BCPD in MSE sense. This paper elucidates that training the teacher model with MSE loss equates to minimizing the MSE between its output and BCPD, aligning with its core responsibility of providing the student with a BCPD estimate closely resembling it in MSE terms. In this respect, through a comprehensive set of experiments, we demonstrate that substituting the conventional teacher trained with cross-entropy loss with one trained using MSE loss in state-of-the-art KD methods consistently boosts the student's accuracy, resulting in improvements of up to 2.6%.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

41 of 55 references.

References · 41

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2019Relational Knowledge Distillationcited by 873
  7. 2019Contrastive Representation Distillationcited by 734
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  9. 2019Similarity-Preserving Knowledge Distillationcited by 507
  10. 2022Decoupled Knowledge Distillationcited by 380
  11. 2019On the Efficacy of Knowledge Distillationcited by 377
  12. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
29 more
  1. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  2. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  3. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  4. 2021Does Knowledge Distillation Really Work?cited by 187
  5. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  6. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  7. 2021Towards Understanding Knowledge Distillationcited by 158
  8. 2018Large scale distributed neural network training through online distillationcited by 144
  9. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  10. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  11. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  12. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  13. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replaycited by 70
  14. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  15. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Expertscited by 50
  16. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  17. 2020All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modelingcited by 42
  18. 2023MetaGCD: Learning to Continually Learn in Generalized Category Discoverycited by 40
  19. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  20. 2023Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalizationcited by 31
  21. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  22. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  23. 2020Subclass Distillationcited by 23
  24. 2024Adapting to Distribution Shift by Visual Domain Prompt Generationcited by 23
  25. 2020Online Knowledge Distillation via Multi-branch Diversity Enhancementcited by 21
  26. 2022Better Supervisory Signals by Observing Learning Pathscited by 21
  27. 2022Weighted Distillation with Unlabeled Examplescited by 12
  28. 2022Toward Student-Oriented Teacher Network Training For Knowledge Distillationcited by 7
  29. 2021On the Efficiency of Subclass Knowledge Distillation in Classification Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptationcited by 14
  2. 2025Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentationcited by 12
  3. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 8
  4. 2025Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillationcited by 5
  5. 2025Towards Undistillable Models by Minimizing Conditional Mutual Informationcited by 3
  6. 2025Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentationcited by 2
  7. 2026Trust the uncertain teacher: distilling dark knowledge via calibrated uncertaintycited by 2
  8. 2024Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Trackingcited by 1
  9. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversitycited by 1
  10. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  11. 2026BrainStack: Neuro-MoE with Functionally Guided Expert Routing for EEG-Based Language Decodingcited by 0
  12. 2026A Unified Revisit of Temperature in Classification-Based Knowledge Distillationcited by 0
1 more
  1. 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 0