mlsanityData to 7 Oct 2026

Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher

arXiv 2010.10090 ↗First posted 20 Oct 2020cs.LG cs.AI stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is a strategy of training a student network with guide of the soft output from a teacher network. It has been a successful method of model compression and knowledge transfer. However, currently knowledge distillation lacks a convincing theoretical understanding. On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency. Based on this metric, we show that for a perfect teacher, a high ratio of teacher's soft labels can be beneficial. Finally, for the case of imperfect teacher, we find that hard labels can correct teacher's wrong prediction, which explains the practice of mixing hard and soft labels.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 10; 2024: 6; 2025: 7; 2026: 4.

33 papers.

Sort:

By area:

  1. 2022Linkless Link Prediction via Relational Distillationcited by 39
  2. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  3. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  4. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 20
  5. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  6. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  7. 2021Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teachingcited by 18
  8. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  9. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  10. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 9
  11. 2023Learning From Biased Soft Labelscited by 8
  12. 2025Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearningcited by 8
21 more
  1. 2023DOT: A Distillation-Oriented Trainercited by 7
  2. 2023Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clusteringcited by 7
  3. 2023Random Teachers are Good Teacherscited by 6
  4. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  5. 2026Channel Location Constrains the Auditability of Subliminal Learningcited by 5
  6. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  7. 2024Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatchcited by 4
  8. 2024Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasetscited by 3
  9. 2023Frameless Graph Knowledge Distillationcited by 2
  10. 2025Neural Tangent Knowledge Distillation for Optical Convolutional Networkscited by 2
  11. 2022On effects of Knowledge Distillation on Transfer Learningcited by 1
  12. 2024Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Modelscited by 1
  13. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  14. 2025LiteLMGuard: Seamless and Lightweight On-Device Prompt Filtering for Safeguarding Small Language Models against Quantization-induced Risks and Vulnerabilitiescited by 1
  15. 2025Revisiting Knowledge Distillation: The Hidden Role of Dataset Sizecited by 1
  16. 2022Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexitycited by 0
  17. 2025A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognitioncited by 0
  18. 2025Enriching Knowledge Distillation with Intra-Class Contrastive Learningcited by 0
  19. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic Viewcited by 0
  20. 2026Optimal Self-Distillation for Rectified Flow via Linear Probingcited by 0
  21. 2026Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performancecited by 0