The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
Self-distillation (SD), a technique where a model improves itself using its own predictions, has attracted attention as a simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tuned multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, we employ the replica method from statistical physics. Our findings reveal that the primary driver of SD's performance improvement is denoising through hard pseudo-labels, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: early stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate our theoretical findings derived from our toy model, we conduct additional experiments on CIFAR-10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing our understanding and application of SD in noisy settings.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 37 of its 61 references that are in the map. First posted on arXiv 27 Jan 2025.
Builds on
37 of 61 references.
Closest ancestors · 6
Same direction, newest first
- 2024A Theoretical Analysis of Soft-Label vs Hard-Label Training in Neural Networks12 Dec
- 2024Understanding the Gains from Repeated Self-Distillation5 Jul
- 2023Understanding Self-Distillation in the Presence of Label Noise30 Jan
- 2021Towards Understanding Knowledge Distillation27 May
- 2021Undistillable: Making A Nasty Teacher That CANNOT teach students16 May
- 2021Learning curves of generic features maps for realistic datasets with a teacher-student model16 Feb
Other references · 31
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Meta-Learning in Neural Networks: A Surveycited by 950
- 2018Born Again Neural Networkscited by 546
- 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
- 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
- 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
- 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
- 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
- 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
- 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
- 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
- 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
19 more
- 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
- 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
- 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
- 2018Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detectioncited by 37
- 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
- 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
- 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 26
- 2020Solvable Model for Inheriting the Regularization through Knowledge Distillationcited by 16
- 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
- 2019Asymptotic Bayes risk for Gaussian mixture in a semi-supervised settingcited by 15
- 2022Bias-inducing geometries: an exactly solvable data model with fairness implicationscited by 9
- 2022The Role of Pseudo-labels in Self-training Linear Classifiers on High-dimensional Gaussian Mixture Datacited by 7
- 2023Regularized Linear Regression for Binary Classificationcited by 7
- 2023Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplingscited by 6
- 2024Retraining with Predicted Hard Labels Provably Increases Model Accuracycited by 5
- 2024A replica analysis of under-baggingcited by 4
- 2024Unifying AMP Algorithms for Rotationally-Invariant Modelscited by 4
- 2024Asymptotic Dynamics of Alternating Minimization for Bilinear Regressioncited by 3
- 2024Adversarial Sparse Teacher: Defense Against Distillation-Based Model Stealing Attacks Using Adversarial Examplescited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels
- 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting
- 2026Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
- 2022Self-Distillation from the Last Mini-Batch for Consistency Regularization
- 2026Data Density Scaling Laws for Image Self-Distillation
- 2023Self-Distillation for Gaussian Process Regression and Classification
- 2023Self-discipline on multiple channels
- 2023Bayesian Optimization Meets Self-Distillation