# Debiased Learning from Naturally Imbalanced Pseudo-Labels

arXiv: https://arxiv.org/abs/2201.01490 · HTML: https://mlsanity.com/paper/2201.01490/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 72 references that are in the map.
- First posted on arXiv 5 Jan 2022.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)

## Built on by (most cited first, 30 of 85)

- [SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning](https://mlsanity.com/paper/2301.10921/) (2023)
- [Parametric Classification for Generalized Category Discovery: A Baseline Study](https://mlsanity.com/paper/2211.11727/) (2022)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [DELTA: degradation-free fully test-time adaptation](https://mlsanity.com/paper/2301.13018/) (2023)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [The Neglected Tails in Vision-Language Models](https://mlsanity.com/paper/2401.12425/) (2024)
- [ProMix: Combating Label Noise via Maximizing Clean Sample Utility](https://mlsanity.com/paper/2207.10276/) (2022)
- [Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2310.11320/) (2023)
- [Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.02899/) (2024)
- [Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning](https://mlsanity.com/paper/2306.01669/) (2023)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting](https://mlsanity.com/paper/2410.19294/) (2024)
- [Solving the Catastrophic Forgetting Problem in Generalized Category Discovery](https://mlsanity.com/paper/2501.05272/) (2025)
- [DHC: Dual-debiased Heterogeneous Co-training Framework for Class-imbalanced Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2307.11960/) (2023)
- [ChatGPT-Powered Hierarchical Comparisons for Image Classification](https://mlsanity.com/paper/2311.00206/) (2023)
- [Semi-supervised learning made simple with self-supervised clustering](https://mlsanity.com/paper/2306.07483/) (2023)
- [TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training](https://mlsanity.com/paper/2312.12828/) (2023)
- [Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning](https://mlsanity.com/paper/2405.11756/) (2024)
- [Imbalanced Semi-supervised Learning with Bias Adaptive Classifier](https://mlsanity.com/paper/2207.13856/) (2022)
- [Continual Learning with Evolving Class Ontologies](https://mlsanity.com/paper/2210.04993/) (2022)
- [ProtoCon: Pseudo-label Refinement via Online Clustering and Prototypical Consistency for Efficient Semi-supervised Learning](https://mlsanity.com/paper/2303.13556/) (2023)
- [FedDiv: Collaborative Noise Filtering for Federated Learning with Noisy Labels](https://mlsanity.com/paper/2312.12263/) (2023)
- [Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.06946/) (2024)
- [What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights](https://mlsanity.com/paper/2405.21070/) (2024)
- [An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2211.11086/) (2022)
- [InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2303.07269/) (2023)
- [RESMatch: Referring Expression Segmentation in a Semi-Supervised Manner](https://mlsanity.com/paper/2402.05589/) (2024)
- [DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank](https://mlsanity.com/paper/2310.14577/) (2023)
- [CDMAD: Class-Distribution-Mismatch-Aware Debiasing for Class-Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2403.10391/) (2024)
- [Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations](https://mlsanity.com/paper/2305.12715/) (2023)

## Abstract (arXiv)

Pseudo-labels are confident predictions made on unlabeled target data by a classifier trained on labeled source data. They are widely used for adapting a model to unlabeled data, e.g., in a semi-supervised learning setting. Our key insight is that pseudo-labels are naturally imbalanced due to intrinsic data similarity, even when a model is trained on balanced source data and evaluated on balanced target data. If we address this previously unknown imbalanced classification problem arising from pseudo-labels instead of ground-truth training labels, we could remove model biases towards false majorities created by pseudo-labels. We propose a novel and effective debiased learning method with pseudo-labels, based on counterfactual reasoning and adaptive margins: The former removes the classifier response bias, whereas the latter adjusts the margin of each class according to the imbalance of pseudo-labels. Validated by extensive experimentation, our simple debiased learning delivers significant accuracy gains over the state-of-the-art on ImageNet-1K: 26% for semi-supervised learning with 0.2% annotations and 9% for zero-shot learning. Our code is available at: https://github.com/frank-xwang/debiased-pseudo-labeling.