mlsanityData to 7 Oct 2026

Understanding the Gains from Repeated Self-Distillation

arXiv 2407.04600 ↗First posted 5 Jul 2024cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Self-Distillation is a special type of knowledge distillation where the student model has the same architecture as the teacher model. Despite using the same architecture and the same training data, self-distillation has been empirically observed to improve performance, especially when applied repeatedly. For such a process, there is a fundamental question of interest: How much gain is possible by applying multiple steps of self-distillation? To investigate this relative gain, we propose studying the simple but canonical task of linear regression. Our analysis shows that the excess risk achieved by multi-step self-distillation can significantly improve upon a single step of self-distillation, reducing the excess risk by a factor as large as d, where d is the input dimension. Empirical results on regression tasks from the UCI repository show a reduction in the learnt model's risk (MSE) by up to 47%.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

32 of 47 references.

References · 32

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2020Knowledge Distillation: A Surveycited by 1,578
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  7. 2017Deep Mutual Learningcited by 621
  8. 2018Born Again Neural Networkscited by 539
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  10. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  11. 2013Do Deep Nets Really Need to be Deep?cited by 466
  12. 2015Policy Distillationcited by 414
20 more
  1. 2019Variational Information Distillation for Knowledge Transfercited by 321
  2. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  3. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  4. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  5. 2015Unifying distillation and privileged informationcited by 218
  6. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  7. 2017Learning from Noisy Labels with Distillationcited by 160
  8. 2021Towards Understanding Knowledge Distillationcited by 158
  9. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  10. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  11. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  12. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  13. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  14. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  15. 2018Improving the Interpretability of Deep Neural Networks with Knowledge Distillationcited by 37
  16. 2023Label Poisoning is All You Needcited by 27
  17. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  18. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  19. 2022Eliminating Backdoor Triggers for Deep Neural Networks Using Attention Relation Graph Distillationcited by 18
  20. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  2. 2025Distillation Scaling Lawscited by 59
  3. 2024Progressive distillation induces an implicit curriculumcited by 15
  4. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  5. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  6. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  7. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  8. 2026Knowledge Distillation for mmWave Beam Prediction Using Sub-6 GHz Channelscited by 1
  9. 2025Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passingcited by 0
  10. 2025When are radiology reports useful for training medical image classifiers?cited by 0
  11. 2025Understanding the Gain from Data Filtering in Multimodal Contrastive Learningcited by 0
  12. 2026Smoothing the Black-Box: Signed-Distance Supervision for Black-Box Model Copyingcited by 0
2 more
  1. 2026Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Modelscited by 0
  2. 2026Optimal Self-Distillation for Rectified Flow via Linear Probingcited by 0