Data to 14 Oct 2026

Self-Distillation Amplifies Regularization in Hilbert Space

arXiv 2002.05715 ↗First posted 13 Feb 2020cs.LG stat.MLcited by 173

Not on the map. Why?

Cited by 78 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of this phenomenon. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to ℓ2 regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 36; 2022: 26; 2023: 31; 2024: 33; 2025: 23; 2026: 24.

173 papers.

Sort:

By area:

2021R-Drop: Regularized Dropout for Neural Networkscited by 202
2021Does Knowledge Distillation Really Work?cited by 190
2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
2024Model Collapse Demystified: The Case of Regressioncited by 82
2022Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillationcited by 62
2021Essentials for Class Incremental Learningcited by 57
2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57

38 more

2021Localization Distillation for Dense Object Detectioncited by 55
2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
2025The Best Instruction-Tuning Data are Those That Fitcited by 52
2021What can linearized neural networks actually say about generalization?cited by 49
2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39
2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 37
2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
2021Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognitioncited by 30
2022Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labelscited by 30
2021Knowledge Distillation as Semiparametric Inferencecited by 29
2025Pre-training under infinite computecited by 29
2022Fast and Memory-Efficient Network Towards Efficient Image Super-Resolutioncited by 27
2022Towards Federated Learning against Noisy Labels via Local Self-Regularizationcited by 26
2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
2024How to Synthesize Text Data without Model Collapse?cited by 25
2021Anchor-based Plain Net for Mobile Image Super-Resolutioncited by 24
2022Revisiting Self-Distillationcited by 24
2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
2022Geometric Knowledge Distillation: Topology Compression for Graph Neural Networkscited by 22
2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 22
2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
2021Teacher's pet: understanding and mitigating biases in distillationcited by 21
2021Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognitioncited by 21
2023Understanding Self-Distillation in the Presence of Label Noisecited by 21
2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
2021Multi-label Iterated Learning for Image Classification with Label Ambiguitycited by 19
2022Localization Distillation for Object Detectioncited by 19
2022Overcoming the Spectral Bias of Neural Value Approximationcited by 19
2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
2021Churn Reduction via Distillationcited by 17
2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17

50 most cited of 173 All 173 (JSON)