# Better Self-training for Image Classification through Self-supervision

arXiv: https://arxiv.org/abs/2109.00778 · HTML: https://mlsanity.com/paper/2109.00778/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [An open access repository of images on plant health to enable the development of mobile disease diagnostics](https://mlsanity.com/paper/1511.08060/) (2015)

## Built on by (most cited first, 9 of 9)

- [Masked Unsupervised Self-training for Label-free Image Classification](https://mlsanity.com/paper/2206.02967/) (2022)
- [Revisiting DETR Pre-training for Object Detection](https://mlsanity.com/paper/2308.01300/) (2023)
- [DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models](https://mlsanity.com/paper/2408.08855/) (2024)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels](https://mlsanity.com/paper/2404.10146/) (2024)
- [Towards Label-Efficient Incremental Learning: A Survey](https://mlsanity.com/paper/2302.00353/) (2023)
- [3D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinement](https://mlsanity.com/paper/2510.17875/) (2025)
- [Contrastive Credibility Propagation for Reliable Semi-Supervised Learning](https://mlsanity.com/paper/2211.09929/) (2022)
- [Cordyceps@LT-EDI: Depression Detection with Reddit and Self-training](https://mlsanity.com/paper/2310.01418/) (2023)

## Abstract (arXiv)

Self-training is a simple semi-supervised learning approach: Unlabelled examples that attract high-confidence predictions are labelled with their predictions and added to the training set, with this process being repeated multiple times. Recently, self-supervision -- learning without manual supervision by solving an automatically-generated pretext task -- has gained prominence in deep learning. This paper investigates three different ways of incorporating self-supervision into self-training to improve accuracy in image classification: self-supervision as pretraining only, self-supervision performed exclusively in the first iteration of self-training, and self-supervision added to every iteration of self-training. Empirical results on the SVHN, CIFAR-10, and PlantVillage datasets, using both training from scratch, and Imagenet-pretrained weights, show that applying self-supervision only in the first iteration of self-training can greatly improve accuracy, for a modest increase in computation time.