Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing
Not on the map. Why?
Abstract (arXiv)
Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance. While prior works have demonstrated the benefits of specific heuristic retraining schemes, the question of how to optimally combine the model's predictions and the provided labels remains largely open. This paper addresses this fundamental question for binary classification tasks. We develop a principled framework based on approximate message passing (AMP) to analyze iterative retraining procedures for two ground truth settings: Gaussian mixture model (GMM) and generalized linear model (GLM). Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels, which when used to retrain the same model, minimizes its prediction error. We also quantify the performance of this optimal retraining strategy over multiple rounds. We complement our theoretical results by proposing a practically usable version of the theoretically-optimal aggregator function for linear probing with the cross-entropy loss, and demonstrate its superiority over baseline methods in the high label noise regime.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 21 May 2025.
Builds on
31 of 46 references.
References · 31
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
- 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
- 2018Born Again Neural Networkscited by 546
- 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
- 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
- 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
- 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
- 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
- 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
- 2017Learning from Noisy Labels with Distillationcited by 161
19 more
- 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 149
- 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
- 2019Deep Self-Learning From Noisy Labelscited by 122
- 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
- 2022Self-Training: A Surveycited by 86
- 2020Self-training Avoids Using Spurious Features Under Domain Shiftcited by 69
- 2021A unifying tutorial on Approximate Message Passingcited by 50
- 2017Estimation of Low-Rank Matrices via Approximate Message Passingcited by 41
- 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 28
- 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
- 2020Approximate Message Passing with Spectral Initialization for Generalized Linear Modelscited by 22
- 2023Understanding Self-Distillation in the Presence of Label Noisecited by 21
- 2022How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysiscited by 19
- 2016Message-passing algorithms for synchronization problems over compact groupscited by 18
- 2024Understanding the Gains from Repeated Self-Distillationcited by 15
- 2020Statistical and Algorithmic Insights for Semi-supervised Learning with Self-trainingcited by 12
- 2022PARS: Pseudo-Label Aware Robust Sample Selection for Learning with Noisy Labelscited by 7
- 2024Retraining with Predicted Hard Labels Provably Increases Model Accuracycited by 5
- 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.