# Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training

arXiv: https://arxiv.org/abs/2006.11006 · HTML: https://mlsanity.com/paper/2006.11006/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 12 of 12)

- [Unsupervised Multi-source Domain Adaptation Without Access to Source Data](https://mlsanity.com/paper/2104.01845/) (2021)
- [Generate, Annotate, and Learn: NLP with Synthetic Text](https://mlsanity.com/paper/2106.06168/) (2021)
- [Self-training Converts Weak Learners to Strong Learners in Mixture Models](https://mlsanity.com/paper/2106.13805/) (2021)
- [How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis](https://mlsanity.com/paper/2201.08514/) (2022)
- [Retraining with Predicted Hard Labels Provably Increases Model Accuracy](https://mlsanity.com/paper/2406.11206/) (2024)
- [Why the pseudo label based semi-supervised learning algorithm is effective?](https://mlsanity.com/paper/2211.10039/) (2022)
- [Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergence](https://mlsanity.com/paper/2405.00454/) (2024)
- [Sharp-SSL: Selective high-dimensional axis-aligned random projections for semi-supervised learning](https://mlsanity.com/paper/2304.09154/) (2023)
- [Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing](https://mlsanity.com/paper/2505.15195/) (2025)
- [Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models](https://mlsanity.com/paper/2605.17778/) (2026)
- [Non-Convex Optimizations for Machine Learning with Theoretical Guarantee: Robust Matrix Completion and Neural Network Learning](https://mlsanity.com/paper/2306.16557/) (2023)
- [VBFDD-Agent for Electric Vehicle Battery Fault Detection and Diagnosis: Descriptive Text Modeling of Battery Digital Signals](https://mlsanity.com/paper/2605.20742/) (2026)

## Abstract (arXiv)

Self-training is a classical approach in semi-supervised learning which is successfully applied to a variety of machine learning problems. Self-training algorithm generates pseudo-labels for the unlabeled examples and progressively refines these pseudo-labels which hopefully coincides with the actual labels. This work provides theoretical insights into self-training algorithm with a focus on linear classifiers. We first investigate Gaussian mixture models and provide a sharp non-asymptotic finite-sample characterization of the self-training iterations. Our analysis reveals the provable benefits of rejecting samples with low confidence and demonstrates that self-training iterations gracefully improve the model accuracy even if they do get stuck in sub-optimal fixed points. We then demonstrate that regularization and class margin (i.e. separation) is provably important for the success and lack of regularization may prevent self-training from identifying the core features in the data. Finally, we discuss statistical aspects of empirical risk minimization with self-training for general distributions. We show how a purely unsupervised notion of generalization based on self-training based clustering can be formalized based on cluster margin. We then establish a connection between self-training based semi-supervision and the more general problem of learning with heterogenous data and weak supervision.