# In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning

arXiv: https://arxiv.org/abs/2101.06329 · HTML: https://mlsanity.com/paper/2101.06329/

## Key facts

- Cited by 301 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 79 references that are in the map.
- First posted on arXiv 15 Jan 2021.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation](https://mlsanity.com/paper/1907.07034/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)

## Built on by (most cited first, 30 of 301)

- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](https://mlsanity.com/paper/2205.07246/) (2022)
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](https://mlsanity.com/paper/2203.03884/) (2022)
- [Efficient Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2403.18293/) (2024)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [Mutual Consistency Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2109.09960/) (2021)
- [Self-Improvement in Language Models: The Sharpening Mechanism](https://mlsanity.com/paper/2412.01951/) (2024)
- [Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration](https://mlsanity.com/paper/2109.14285/) (2021)
- [UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning](https://mlsanity.com/paper/2203.14542/) (2022)
- [Debiased Self-Training for Semi-Supervised Learning](https://mlsanity.com/paper/2202.07136/) (2022)
- [A Close Look into the Calibration of Pre-trained Language Models](https://mlsanity.com/paper/2211.00151/) (2022)
- [Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances](https://mlsanity.com/paper/2111.00418/) (2021)
- [Better Zero-Shot Reasoning with Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14106/) (2023)
- [Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2212.04090/) (2022)
- [Semi-supervised 3D Object Detection with Proficient Teachers](https://mlsanity.com/paper/2207.12655/) (2022)
- [Structured Sparse R-CNN for Direct Scene Graph Generation](https://mlsanity.com/paper/2106.10815/) (2021)
- [KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2205.06784/) (2022)
- [Learning Fair Classifiers with Partially Annotated Group Labels](https://mlsanity.com/paper/2111.14581/) (2021)
- [Parametric Information Maximization for Generalized Category Discovery](https://mlsanity.com/paper/2212.00334/) (2022)
- [OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2207.02261/) (2022)
- [ACPL: Anti-curriculum Pseudo-labelling for Semi-supervised Medical Image Classification](https://mlsanity.com/paper/2111.12918/) (2021)
- [Continual Test-time Domain Adaptation via Dynamic Sample Selection](https://mlsanity.com/paper/2310.03335/) (2023)
- [Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision](https://mlsanity.com/paper/2201.07929/) (2022)
- [Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation](https://mlsanity.com/paper/2112.00735/) (2021)
- [Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries](https://mlsanity.com/paper/2203.15355/) (2022)
- [BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation](https://mlsanity.com/paper/2210.05174/) (2022)
- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [Uncertainty-Aware Learning Against Label Noise on Imbalanced Datasets](https://mlsanity.com/paper/2207.05471/) (2022)
- [Towards Realistic Semi-Supervised Learning](https://mlsanity.com/paper/2207.02269/) (2022)
- [Learning from Temporal Gradient for Semi-supervised Action Recognition](https://mlsanity.com/paper/2111.13241/) (2021)

## Abstract (arXiv)

The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.