# Why distillation helps: a statistical perspective

arXiv: https://arxiv.org/abs/2005.10419 · HTML: https://mlsanity.com/paper/2005.10419/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 21 of 21)

- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [S-LIME: Stabilized-LIME for Model Explanation](https://mlsanity.com/paper/2106.07875/) (2021)
- [Towards Model Agnostic Federated Learning Using Knowledge Distillation](https://mlsanity.com/paper/2110.15210/) (2021)
- [Knowledge Distillation as Semiparametric Inference](https://mlsanity.com/paper/2104.09732/) (2021)
- [High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws](https://mlsanity.com/paper/2410.18837/) (2024)
- [Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery](https://mlsanity.com/paper/2601.20088/) (2026)
- [Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification](https://mlsanity.com/paper/2208.08741/) (2022)
- [Churn Reduction via Distillation](https://mlsanity.com/paper/2106.02654/) (2021)
- [Knowledge Distillation: Bad Models Can Be Good Role Models](https://mlsanity.com/paper/2203.14649/) (2022)
- [Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR](https://mlsanity.com/paper/2210.05793/) (2022)
- [Robust Active Distillation](https://mlsanity.com/paper/2210.01213/) (2022)
- [Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation](https://mlsanity.com/paper/2101.02375/) (2021)
- [When in Doubt, Summon the Titans: Efficient Inference with Large Models](https://mlsanity.com/paper/2110.10305/) (2021)
- [Distilling Double Descent](https://mlsanity.com/paper/2102.06849/) (2021)
- [Towards the Fundamental Limits of Knowledge Transfer over Finite Domains](https://mlsanity.com/paper/2310.07838/) (2023)
- [DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective](https://mlsanity.com/paper/2507.02911/) (2025)
- [Beyond Real Data: Synthetic Data through the Lens of Regularization](https://mlsanity.com/paper/2510.08095/) (2025)
- [Knowledge Distillation for Oriented Object Detection on Aerial Images](https://mlsanity.com/paper/2206.09796/) (2022)
- [Triplet Knowledge Distillation](https://mlsanity.com/paper/2305.15975/) (2023)
- [Parametric ρ-Norm Scaling Calibration](https://mlsanity.com/paper/2412.15301/) (2024)
- [Scaling Laws for Task-Specific LLM Distillation](https://mlsanity.com/paper/2606.24747/) (2026)

## Abstract (arXiv)

Knowledge distillation is a technique for improving the performance of a simple "student" model by replacing its one-hot training labels with a distribution over labels obtained from a complex "teacher" model. While this simple approach has proven widely effective, a basic question remains unresolved: why does distillation help? In this paper, we present a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques. Our core observation is that the teacher seeks to estimate the underlying (Bayes) class-probability function. Building on this, we establish a fundamental bias-variance tradeoff in the student's objective: this quantifies how approximate knowledge of these class-probabilities can significantly aid learning. Finally, we show how distillation complements existing negative mining techniques for extreme multiclass retrieval, and propose a unified objective which combines these ideas.