# Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning

arXiv: https://arxiv.org/abs/2106.00209 · HTML: https://mlsanity.com/paper/2106.00209/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Learning with Pseudo-Ensembles](https://mlsanity.com/paper/1412.4864/) (2014)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2102.09559/) (2021)
- [M2m: Imbalanced Classification via Major-to-minor Translation](https://mlsanity.com/paper/2004.00431/) (2020)
- [Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspective](https://mlsanity.com/paper/2002.10826/) (2020)
- [Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2007.08844/) (2020)
- [Class-Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2002.06815/) (2020)
- [Mutual Exclusivity Loss for Semi-Supervised Deep Learning](https://mlsanity.com/paper/1606.03141/) (2016)

## Built on by (most cited first, 16 of 16)

- [CoSSL: Co-Learning of Representation and Classifier for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2112.04564/) (2021)
- [Revisiting Consistency Regularization for Semi-Supervised Learning](https://mlsanity.com/paper/2112.05825/) (2021)
- [ImGCL: Revisiting Graph Contrastive Learning on Imbalanced Node Classification](https://mlsanity.com/paper/2205.11332/) (2022)
- [Class-Level Confidence Based 3D Semi-Supervised Learning](https://mlsanity.com/paper/2210.10138/) (2022)
- [Semi-Supervised Graph Imbalanced Regression](https://mlsanity.com/paper/2305.12087/) (2023)
- [An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2211.11086/) (2022)
- [Gradient-based Sampling for Class Imbalanced Semi-supervised Object Detection](https://mlsanity.com/paper/2403.15127/) (2024)
- [Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data](https://mlsanity.com/paper/2306.09697/) (2023)
- [NP-Match: Towards a New Probabilistic Model for Semi-Supervised Learning](https://mlsanity.com/paper/2301.13569/) (2023)
- [Pushing the Envelope for Depth-Based Semi-Supervised 3D Hand Pose Estimation with Consistency Training](https://mlsanity.com/paper/2303.15147/) (2023)
- [Generalized Category Discovery under the Long-Tailed Distribution](https://mlsanity.com/paper/2506.12515/) (2025)
- [Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks](https://mlsanity.com/paper/2209.12157/) (2022)
- [BMB: Balanced Memory Bank for Imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2305.12912/) (2023)
- [Class-Specific Distribution Alignment for Semi-Supervised Medical Image Classification](https://mlsanity.com/paper/2307.15987/) (2023)
- [Exploring Probabilistic Models for Semi-supervised Learning](https://mlsanity.com/paper/2404.04199/) (2024)
- [Imbalanced Semi-Supervised Learning via Label Refinement and Threshold Adjustment](https://mlsanity.com/paper/2407.05370/) (2024)

## Abstract (arXiv)

Semi-Supervised Learning (SSL) has shown its strong ability in utilizing unlabeled data when labeled data is scarce. However, most SSL algorithms work under the assumption that the class distributions are balanced in both training and test sets. In this work, we consider the problem of SSL on class-imbalanced data, which better reflects real-world situations. In particular, we decouple the training of the representation and the classifier, and systematically investigate the effects of different data re-sampling techniques when training the whole network including a classifier as well as fine-tuning the feature extractor only. We find that data re-sampling is of critical importance to learn a good classifier as it increases the accuracy of the pseudo-labels, in particular for the minority classes in the unlabeled data. Interestingly, we find that accurate pseudo-labels do not help when training the feature extractor, rather contrariwise, data re-sampling harms the training of the feature extractor. This finding is against the general intuition that wrong pseudo-labels always harm the model performance in SSL. Based on these findings, we suggest to re-think the current paradigm of having a single data re-sampling strategy and develop a simple yet highly effective Bi-Sampling (BiS) strategy for SSL on class-imbalanced data. BiS implements two different re-sampling strategies for training the feature extractor and the classifier and integrates this decoupled training into an end-to-end framework. In particular, BiS progressively changes the data distribution during training such that in the beginning the feature extractor is trained effectively, while towards the end of the training the data is re-balanced such that the classifier is trained reliably. We benchmark our proposed bi-sampling strategy extensively on popular datasets and achieve state-of-the-art performances.