mlsanityData to 7 Oct 2026

The Rich Get Richer: Disparate Impact of Semi-Supervised Learning

arXiv 2110.06282 ↗First posted 12 Oct 2021cs.LG stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Semi-supervised learning (SSL) has demonstrated its potential to improve the model accuracy for a variety of learning tasks when the high-quality supervised data is severely limited. Although it is often established that the average accuracy for the entire population of data is improved, it is unclear how SSL fares with different sub-populations. Understanding the above question has substantial fairness implications when different sub-populations are defined by the demographic groups that we aim to treat fairly. In this paper, we reveal the disparate impacts of deploying SSL: the sub-population who has a higher baseline accuracy without using SSL (the "rich" one) tends to benefit more from SSL; while the sub-population who suffers from a low baseline accuracy (the "poor" one) might even observe a performance drop after adding the SSL module. We theoretically and empirically establish the above observation for a broad family of SSL algorithms, which either explicitly or implicitly use an auxiliary "pseudo-label". Experiments on a set of image and text classification tasks confirm our claims. We introduce a new metric, Benefit Ratio, and promote the evaluation of the fairness of SSL (Equalized Benefit Ratio). We further discuss how the disparate impact can be mitigated. We hope our paper will alarm the potential pitfall of using SSL and encourage a multifaceted evaluation of future SSL algorithms.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 47 of its 85 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

47 of 85 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  8. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  9. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  10. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  11. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  12. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
35 more
  1. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  2. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  3. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  4. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  5. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  6. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  7. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  8. 2015Semi-Supervised Learning with Ladder Networkscited by 246
  9. 2018Multiaccuracy: Black-Box Post-Processing for Fairness in Classificationcited by 221
  10. 2015Semi-supervised Sequence Learningcited by 198
  11. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  12. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  13. 2020Characterising Bias in Compressed Modelscited by 162
  14. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  15. 2020Minimax Pareto Fairness: A Multi Objective Perspectivecited by 152
  16. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 147
  17. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  18. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 112
  19. 2020A Second-Order Approach to Learning with Instance-Dependent Label Noisecited by 90
  20. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  21. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  22. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  23. 2020Fair Classification with Group-Dependent Label Noisecited by 67
  24. 2018Weakly-Supervised Neural Text Classificationcited by 66
  25. 2021Clusterability as an Alternative to Anchor Points When Learning with Noisy Labelscited by 65
  26. 2020When Optimizing f-divergence is Robust with Label Noisecited by 44
  27. 2019Variational Pretraining for Semi-supervised Text Classificationcited by 41
  28. 2021Self-Supervised Learning for Semi-Supervised Temporal Action Proposalcited by 39
  29. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  30. 2020Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Functioncited by 23
  31. 2021Understanding Instance-Level Label Noise: Disparate Impacts and Treatmentscited by 22
  32. 2021Multiaccurate Proxies for Downstream Fairnesscited by 17
  33. 2021Can Less be More? When Increasing-to-Balancing Label Noise Rates Considered Beneficialcited by 13
  34. 2019Variational Sequential Labelers for Semi-Supervised Learningcited by 11
  35. 2020SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 11; 2023: 10; 2024: 1; 2025: 3; 2026: 1.

28 papers.

Sort:

By area:

  1. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  2. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  3. 2022Pruning has a disparate impact on model accuracycited by 43
  4. 2023T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generationcited by 39
  5. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuningcited by 30
  6. 2022To Aggregate or Not? Learning with Separate Noisy Labelscited by 29
  7. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  8. 2022Understanding Instance-Level Impact of Fairness Constraintscited by 27
  9. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  10. 2022Transferring Fairness under Distribution Shifts via Fair Consistency Regularizationcited by 21
  11. 2022Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributescited by 17
  12. 2022Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Makingcited by 12
16 more
  1. 2024Fairness Without Harm: An Influence-Guided Active Sampling Approachcited by 12
  2. 2022Input-agnostic Certified Group Fairness via Gaussian Parameter Smoothingcited by 11
  3. 2022Don't fear the unlabelled: safe semi-supervised learning via simple debiasingcited by 10
  4. 2023The Importance of Human-Labeled Data in the Era of LLMscited by 8
  5. 2023Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Toolscited by 7
  6. 2021Adaptive Data Debiasing through Bounded Explorationcited by 5
  7. 2023Fairness Improves Learning from Noisily Labeled Long-Tailed Datacited by 5
  8. 2023Holistic Survey of Privacy and Fairness in Machine Learningcited by 4
  9. 2025Noise-Resilient Point-wise Anomaly Detection in Time Series Using Weak Segment Labelscited by 3
  10. 2022Teacher Guided Training: An Efficient Framework for Knowledge Transfercited by 1
  11. 2023On Comparing Fair Classifiers under Data Biascited by 1
  12. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1
  13. 2026Source-Free Domain Adaptation with Vision-Language Priorcited by 1
  14. 2023On the Cause of Unfairness: A Training Sample Perspectivecited by 0
  15. 2025Adaptive Bounded Exploration and Intermediate Actions for Data Debiasingcited by 0
  16. 2025Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Modelscited by 0