mlsanityData to 7 Oct 2026

Knowledge Distillation as Semiparametric Inference

arXiv 2104.09732 ↗First posted 20 Apr 2021stat.ML cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements -- cross-fitting and loss correction -- to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 42 of its 75 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

42 of 75 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2016Learning without Forgettingcited by 2,465
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2019When Does Label Smoothing Help?cited by 985
  10. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  11. 2013Network In Networkcited by 600
  12. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 562
30 more
  1. 2018Born Again Neural Networkscited by 539
  2. 2013Do Deep Nets Really Need to be Deep?cited by 466
  3. 2019On the Efficacy of Knowledge Distillationcited by 377
  4. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  5. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  6. 2015Unifying distillation and privileged informationcited by 218
  7. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  8. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  9. 2019Orthogonal Statistical Learningcited by 150
  10. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  11. 2019Adversarially Robust Distillationcited by 124
  12. 2018Does data interpolation contradict statistical optimality?cited by 95
  13. 2020Understanding and Improving Knowledge Distillationcited by 83
  14. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  15. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 65
  16. 2017Ensemble Distillation for Neural Machine Translationcited by 39
  17. 2018Private Model Compression via Knowledge Distillationcited by 37
  18. 2018De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representerscited by 34
  19. 2020Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillationcited by 27
  20. 2020Why distillation helps: a statistical perspectivecited by 21
  21. 2018Attention-Guided Answer Distillation for Machine Reading Comprehensioncited by 18
  22. 2018Knowledge Distillation with Feature Maps for Image Classificationcited by 17
  23. 2019White-to-Black: Efficient Distillation of Black-Box Adversarial Attackscited by 14
  24. 2016Knowledge Distillation for Small-footprint Highway Networkscited by 11
  25. 2019Information-Theoretic Understanding of Population Risk Improvement with Model Compressioncited by 11
  26. 2015Distilling Word Embeddings: An Encoding Approachcited by 7
  27. 2018Teacher-Student Compression with Generative Adversarial Networkscited by 7
  28. 2018Regularized Orthogonal Machine Learning for Nonlinear Semiparametric Modelscited by 6
  29. 2017Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verificationcited by 3
  30. 2017MLBench: How Good Are Machine Learning Clouds for Binary Classification Tasks on Structured Data?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 10; 2024: 7; 2025: 4; 2026: 2.

29 papers.

Sort:

By area:

  1. 2023LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Versioncited by 49
  2. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  3. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  4. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 20
  5. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  6. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  7. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  8. 2024How to Train the Teacher Model for Effective Knowledge Distillationcited by 13
  9. 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
  10. 2023Learning From Biased Soft Labelscited by 8
  11. 2022Toward Student-Oriented Teacher Network Training For Knowledge Distillationcited by 7
  12. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
17 more
  1. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  2. 2024Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labelscited by 4
  3. 2024Automated Efficient Estimation using Monte Carlo Efficient Influence Functionscited by 4
  4. 2024Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatchcited by 4
  5. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Datacited by 4
  6. 2023Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applicationscited by 2
  7. 2023Parallel Neurosymbolic Integration with Concordiacited by 2
  8. 2023Transparency challenges in policy evaluation with causal machine learning -- improving usability and accountabilitycited by 2
  9. 2022Selective Cross-Task Distillationcited by 1
  10. 2025Preference Learning with Response Time: Robust Losses and Guaranteescited by 1
  11. 2022Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexitycited by 0
  12. 2022Scalable Regularization of Scene Graph Generation Models using Symbolic Theoriescited by 0
  13. 2023Confidence Ranking for CTR Predictioncited by 0
  14. 2024Distilling interpretable causal trees from causal forestscited by 0
  15. 2025ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillationcited by 0
  16. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  17. 2026SelfCue: Making a 3D CT Report Generator Say What It Already Knowscited by 0