# Enhancing Self-Training Methods

arXiv: https://arxiv.org/abs/2301.07294 · HTML: https://mlsanity.com/paper/2301.07294/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 18 Jan 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [An Overview of Deep Semi-Supervised Learning](https://mlsanity.com/paper/2006.05278/) (2020)
- [There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average](https://mlsanity.com/paper/1806.05594/) (2018)
- [Class-Aware Contrastive Semi-Supervised Learning](https://mlsanity.com/paper/2203.02261/) (2022)
- [A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification](https://mlsanity.com/paper/2104.00679/) (2021)
- [Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup](https://mlsanity.com/paper/2111.15454/) (2021)
- [Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data](https://mlsanity.com/paper/2110.13185/) (2021)

## Built on by (most cited first, 2 of 2)

- [Metric-DST: Mitigating Selection Bias Through Diversity-Guided Semi-Supervised Metric Learning](https://mlsanity.com/paper/2411.18442/) (2024)
- [Enhancing Science Classroom Discourse Analysis through Joint Multi-Task Learning for Reasoning-Component Classification](https://mlsanity.com/paper/2604.21137/) (2026)

## Abstract (arXiv)

Semi-supervised learning approaches train on small sets of labeled data along with large sets of unlabeled data. Self-training is a semi-supervised teacher-student approach that often suffers from the problem of "confirmation bias" that occurs when the student model repeatedly overfits to incorrect pseudo-labels given by the teacher model for the unlabeled data. This bias impedes improvements in pseudo-label accuracy across self-training iterations, leading to unwanted saturation in model performance after just a few iterations. In this work, we describe multiple enhancements to improve the self-training pipeline to mitigate the effect of confirmation bias. We evaluate our enhancements over multiple datasets showing performance gains over existing self-training design choices. Finally, we also study the extendability of our enhanced approach to Open Set unlabeled data (containing classes not seen in labeled data).