# Self-Training: A Survey

arXiv: https://arxiv.org/abs/2202.12040 · HTML: https://mlsanity.com/paper/2202.12040/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 136 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 30 of 43)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptation](https://mlsanity.com/paper/1803.11365/) (2018)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Asymmetric Tri-training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1702.08400/) (2017)
- [Predicting What You Already Know Helps: Provable Self-Supervised Learning](https://mlsanity.com/paper/2008.01064/) (2020)
- [Understanding Self-Training for Gradual Domain Adaptation](https://mlsanity.com/paper/2002.11361/) (2020)
- [MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection](https://mlsanity.com/paper/2104.01633/) (2021)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Enabling Language Models to Fill in the Blanks](https://mlsanity.com/paper/2005.05339/) (2020)
- [Cycle Self-Training for Domain Adaptation](https://mlsanity.com/paper/2103.03571/) (2021)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [Recovery Guarantees for One-hidden-layer Neural Networks](https://mlsanity.com/paper/1706.03175/) (2017)
- [Multi-Task Self-Training for Learning General Representations](https://mlsanity.com/paper/2108.11353/) (2021)
- [Self-training Avoids Using Spurious Features Under Domain Shift](https://mlsanity.com/paper/2006.10032/) (2020)
- [Corrupted Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2202.03382/) (2022)

## Built on by (most cited first, 30 of 86)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data](https://mlsanity.com/paper/2502.06737/) (2025)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [West-of-N: Synthetic Preferences for Self-Improving Reward Models](https://mlsanity.com/paper/2401.12086/) (2024)
- [Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation](https://mlsanity.com/paper/2501.04167/) (2025)
- [Balancing Logit Variation for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2306.02061/) (2023)
- [Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.18248/) (2024)
- [STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training](https://mlsanity.com/paper/2412.00161/) (2024)
- [A Review of Pseudo-Labeling for Computer Vision](https://mlsanity.com/paper/2408.07221/) (2024)
- [Rethinking Chain-of-Thought from the Perspective of Self-Training](https://mlsanity.com/paper/2412.10827/) (2024)
- [From Memorization to Creativity: LLM as a Designer of Novel Neural Architectures](https://mlsanity.com/paper/2601.02997/) (2026)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [OntoTune: Ontology-Driven Self-training for Aligning Large Language Models](https://mlsanity.com/paper/2502.05478/) (2025)
- [Fraud Dataset Benchmark and Applications](https://mlsanity.com/paper/2208.14417/) (2022)
- [Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias](https://mlsanity.com/paper/2310.14814/) (2023)
- [LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs](https://mlsanity.com/paper/2405.11162/) (2024)
- [Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts](https://mlsanity.com/paper/2305.03237/) (2023)
- [Dual Classification Head Self-training Network for Cross-scene Hyperspectral Image Classification](https://mlsanity.com/paper/2502.17879/) (2025)
- [Revisiting semi-supervised learning in the era of foundation models](https://mlsanity.com/paper/2503.09707/) (2025)
- [Unsupervised Prototype Adapter for Vision-Language Models](https://mlsanity.com/paper/2308.11507/) (2023)
- [Retraining with Predicted Hard Labels Provably Increases Model Accuracy](https://mlsanity.com/paper/2406.11206/) (2024)
- [Voxels Intersecting along Orthogonal Levels Attention U-Net for Intracerebral Haemorrhage Segmentation in Head CT](https://mlsanity.com/paper/2208.06313/) (2022)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [ST-SAM: SAM-Driven Self-Training Framework for Semi-Supervised Camouflaged Object Detection](https://mlsanity.com/paper/2507.23307/) (2025)
- [Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling](https://mlsanity.com/paper/2302.08659/) (2023)
- [Domain-Adaptive Full-Face Gaze Estimation via Novel-View-Synthesis and Feature Disentanglement](https://mlsanity.com/paper/2305.16140/) (2023)
- [Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding](https://mlsanity.com/paper/2310.13022/) (2023)
- [A Review of Machine Learning Methods Applied to Video Analysis Systems](https://mlsanity.com/paper/2312.05352/) (2023)
- [Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shift](https://mlsanity.com/paper/2401.08909/) (2024)

## Abstract (arXiv)

Semi-supervised algorithms aim to learn prediction functions from a small set of labeled observations and a large set of unlabeled observations. Because this framework is relevant in many applications, they have received a lot of interest in both academia and industry. Among the existing techniques, self-training methods have undoubtedly attracted greater attention in recent years. These models are designed to find the decision boundary on low density regions without making additional assumptions about the data distribution, and use the unsigned output score of a learned classifier, or its margin, as an indicator of confidence. The working principle of self-training algorithms is to learn a classifier iteratively by assigning pseudo-labels to the set of unlabeled training samples with a margin greater than a certain threshold. The pseudo-labeled examples are then used to enrich the labeled training data and to train a new classifier in conjunction with the labeled training set. In this paper, we present self-training methods for binary and multi-class classification; as well as their variants and two related approaches, namely consistency-based approaches and transductive learning. We examine the impact of significant self-training features on various methods, using different general and image classification benchmarks, and we discuss our ideas for future research in self-training. To the best of our knowledge, this is the first thorough and complete survey on this subject.