# Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher

arXiv: https://arxiv.org/abs/2010.10090 · HTML: https://mlsanity.com/paper/2010.10090/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 30 of 33)

- [Linkless Link Prediction via Relational Distillation](https://mlsanity.com/paper/2210.05801/) (2022)
- [Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation](https://mlsanity.com/paper/2305.05010/) (2023)
- [Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again](https://mlsanity.com/paper/2210.04427/) (2022)
- [Teacher's pet: understanding and mitigating biases in distillation](https://mlsanity.com/paper/2106.10494/) (2021)
- [Understanding Self-Distillation in the Presence of Label Noise](https://mlsanity.com/paper/2301.13304/) (2023)
- [Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching](https://mlsanity.com/paper/2112.09459/) (2021)
- [Supervision Complexity and its Role in Knowledge Distillation](https://mlsanity.com/paper/2301.12245/) (2023)
- [On student-teacher deviations in distillation: does it pay to disobey?](https://mlsanity.com/paper/2301.12923/) (2023)
- [Provable Weak-to-Strong Generalization via Benign Overfitting](https://mlsanity.com/paper/2410.04638/) (2024)
- [Feature learning as alignment: a structural property of gradient descent in non-linear neural networks](https://mlsanity.com/paper/2402.05271/) (2024)
- [Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearning](https://mlsanity.com/paper/2501.16663/) (2025)
- [Learning From Biased Soft Labels](https://mlsanity.com/paper/2302.08155/) (2023)
- [DOT: A Distillation-Oriented Trainer](https://mlsanity.com/paper/2307.08436/) (2023)
- [Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering](https://mlsanity.com/paper/2307.11030/) (2023)
- [Channel Location Constrains the Auditability of Subliminal Learning](https://mlsanity.com/paper/2606.22019/) (2026)
- [Random Teachers are Good Teachers](https://mlsanity.com/paper/2302.12091/) (2023)
- [Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers](https://mlsanity.com/paper/2404.09326/) (2024)
- [Towards the Fundamental Limits of Knowledge Transfer over Finite Domains](https://mlsanity.com/paper/2310.07838/) (2023)
- [Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch](https://mlsanity.com/paper/2405.13078/) (2024)
- [Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets](https://mlsanity.com/paper/2410.22184/) (2024)
- [Frameless Graph Knowledge Distillation](https://mlsanity.com/paper/2307.06631/) (2023)
- [The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model](https://mlsanity.com/paper/2501.16226/) (2025)
- [Neural Tangent Knowledge Distillation for Optical Convolutional Networks](https://mlsanity.com/paper/2508.08421/) (2025)
- [On effects of Knowledge Distillation on Transfer Learning](https://mlsanity.com/paper/2210.09668/) (2022)
- [Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Models](https://mlsanity.com/paper/2406.05364/) (2024)
- [LiteLMGuard: Seamless and Lightweight On-Device Prompt Filtering for Safeguarding Small Language Models against Quantization-induced Risks and Vulnerabilities](https://mlsanity.com/paper/2505.05619/) (2025)
- [Revisiting Knowledge Distillation: The Hidden Role of Dataset Size](https://mlsanity.com/paper/2510.15516/) (2025)
- [On the Generalization of Knowledge Distillation: An Information-Theoretic View](https://mlsanity.com/paper/2605.13143/) (2026)
- [Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance](https://mlsanity.com/paper/2608.29472/) (2026)
- [Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity](https://mlsanity.com/paper/2207.00751/) (2022)

## Abstract (arXiv)

Knowledge distillation is a strategy of training a student network with guide of the soft output from a teacher network. It has been a successful method of model compression and knowledge transfer. However, currently knowledge distillation lacks a convincing theoretical understanding. On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency. Based on this metric, we show that for a perfect teacher, a high ratio of teacher's soft labels can be beneficial. Finally, for the case of imperfect teacher, we find that hard labels can correct teacher's wrong prediction, which explains the practice of mixing hard and soft labels.