# Don't fear the unlabelled: safe semi-supervised learning via simple debiasing

arXiv: https://arxiv.org/abs/2203.07512 · HTML: https://mlsanity.com/paper/2203.07512/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 87 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2102.03526/) (2021)
- [The Rich Get Richer: Disparate Impact of Semi-Supervised Learning](https://mlsanity.com/paper/2110.06282/) (2021)
- [How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis](https://mlsanity.com/paper/2201.08514/) (2022)
- [An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift](https://mlsanity.com/paper/2202.12123/) (2022)
- [Contrastive Pessimistic Likelihood Estimation for Semi-Supervised Classification](https://mlsanity.com/paper/1503.00269/) (2015)
- [Kernel Machines With Missing Responses](https://mlsanity.com/paper/1806.02865/) (2018)
- [On missing label patterns in semi-supervised learning](https://mlsanity.com/paper/1904.02883/) (2019)
- [Safe Semi-Supervised Learning of Sum-Product Networks](https://mlsanity.com/paper/1710.03444/) (2017)

## Built on by (most cited first, 10 of 10)

- [Prediction De-Correlated Inference: A safe approach for post-prediction inference](https://mlsanity.com/paper/2312.06478/) (2023)
- [Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning](https://mlsanity.com/paper/2410.11206/) (2024)
- [Revisiting Active Sequential Prediction-Powered Mean Estimation](https://mlsanity.com/paper/2604.18569/) (2026)
- [Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data](https://mlsanity.com/paper/2512.05456/) (2025)
- [Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection](https://mlsanity.com/paper/2409.11653/) (2024)
- [Missing Data in Signal Processing and Machine Learning: Models, Methods and Modern Approaches](https://mlsanity.com/paper/2506.01696/) (2025)
- [Theory-inspired Label Shift Adaptation via Aligned Distribution Mixture](https://mlsanity.com/paper/2411.02047/) (2024)
- [FixCLR: Negative-Class Contrastive Learning for Semi-Supervised Domain Generalization](https://mlsanity.com/paper/2506.20841/) (2025)
- [ModSSC: A Modular Framework for Semi-Supervised Classification on Heterogeneous Data](https://mlsanity.com/paper/2512.13228/) (2025)
- [Avoiding Structural Failure Modes in Tabular Fair SSL: Online Primal-Dual Allocation under Confidence Gating](https://mlsanity.com/paper/2605.16446/) (2026)

## Abstract (arXiv)

Semi-supervised learning (SSL) provides an effective means of leveraging unlabelled data to improve a model performance. Even though the domain has received a considerable amount of attention in the past years, most methods present the common drawback of lacking theoretical guarantees. Our starting point is to notice that the estimate of the risk that most discriminative SSL methods minimise is biased, even asymptotically. This bias impedes the use of standard statistical learning theory and can hurt empirical performance. We propose a simple way of removing the bias. Our debiasing approach is straightforward to implement and applicable to most deep SSL methods. We provide simple theoretical guarantees on the trustworthiness of these modified methods, without having to rely on the strong assumptions on the data distribution that SSL theory usually requires. In particular, we provide generalisation error bounds for the proposed methods. We evaluate debiased versions of different existing SSL methods, such as the Pseudo-label method and Fixmatch, and show that debiasing can compete with classic deep SSL techniques in various settings by providing better calibrated models. Additionally, we provide a theoretical explanation of the intuition of the popular SSL methods.