# Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients

arXiv: https://arxiv.org/abs/2009.13447 · HTML: https://mlsanity.com/paper/2009.13447/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2020.

## Built on by (most cited first, 30 of 34)

- [ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2110.10368/) (2021)
- [Measuring axiomatic soundness of counterfactual image models](https://mlsanity.com/paper/2303.01274/) (2023)
- [Bias Mimicking: A Simple Sampling Approach for Bias Mitigation](https://mlsanity.com/paper/2209.15605/) (2022)
- [RLSbench: Domain Adaptation Under Relaxed Label Shift](https://mlsanity.com/paper/2302.03020/) (2023)
- [Optimising Equal Opportunity Fairness in Model Training](https://mlsanity.com/paper/2205.02393/) (2022)
- [Mitigating Dataset Bias by Using Per-sample Gradient](https://mlsanity.com/paper/2205.15704/) (2022)
- [A Theoretical Analysis of the Learning Dynamics under Class Imbalance](https://mlsanity.com/paper/2207.00391/) (2022)
- [Scalarization for Multi-Task and Multi-Domain Learning at Scale](https://mlsanity.com/paper/2310.08910/) (2023)
- [Few-Shot Learning with Class Imbalance](https://mlsanity.com/paper/2101.02523/) (2021)
- [Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling](https://mlsanity.com/paper/2410.03735/) (2024)
- [FIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced Data](https://mlsanity.com/paper/2206.02792/) (2022)
- [FAST: Improving Controllability for Text Generation with Feedback Aware Self-Training](https://mlsanity.com/paper/2210.03167/) (2022)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models](https://mlsanity.com/paper/2401.03695/) (2024)
- [Diffusion-Jump GNNs: Homophiliation via Learnable Metric Filters](https://mlsanity.com/paper/2306.16976/) (2023)
- [Learning with Noisy Labels: Interconnection of Two Expectation-Maximizations](https://mlsanity.com/paper/2401.04390/) (2024)
- [Dirichlet-based Per-Sample Weighting by Transition Matrix for Noisy Label Learning](https://mlsanity.com/paper/2403.02690/) (2024)
- [Gradient Extrapolation for Debiased Representation Learning](https://mlsanity.com/paper/2503.13236/) (2025)
- [Causal Scene BERT: Improving object detection by searching for challenging groups of data](https://mlsanity.com/paper/2202.03651/) (2022)
- [Comprehensive Validation on Reweighting Samples for Bias Mitigation via AIF360](https://mlsanity.com/paper/2312.12560/) (2023)
- [How does overparametrization affect performance on minority groups?](https://mlsanity.com/paper/2206.03515/) (2022)
- [Mitigating Spurious Correlations via Disagreement Probability](https://mlsanity.com/paper/2411.01757/) (2024)
- [Underage Detection through a Multi-Task and MultiAge Approach for Screening Minors in Unconstrained Imagery](https://mlsanity.com/paper/2506.10689/) (2025)
- [CW-ERM: Improving Autonomous Driving Planning with Closed-loop Weighted Empirical Risk Minimization](https://mlsanity.com/paper/2210.02174/) (2022)
- [Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Sampler](https://mlsanity.com/paper/2406.13871/) (2024)
- [Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples](https://mlsanity.com/paper/2512.22874/) (2025)
- [Combining resampling and reweighting for faithful stochastic optimization](https://mlsanity.com/paper/2105.14694/) (2021)
- [Variational Actor-Critic Algorithms](https://mlsanity.com/paper/2108.01215/) (2021)
- [Alpha and Prejudice: Improving α-sized Worst-case Fairness via Intrinsic Reweighting](https://mlsanity.com/paper/2411.03068/) (2024)
- [Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness](https://mlsanity.com/paper/2504.14882/) (2025)

## Abstract (arXiv)

A data set sampled from a certain population is biased if the subgroups of the population are sampled at proportions that are significantly different from their underlying proportions. Training machine learning models on biased data sets requires correction techniques to compensate for the bias. We consider two commonly-used techniques, resampling and reweighting, that rebalance the proportions of the subgroups to maintain the desired objective function. Though statistically equivalent, it has been observed that resampling outperforms reweighting when combined with stochastic gradient algorithms. By analyzing illustrative examples, we explain the reason behind this phenomenon using tools from dynamical stability and stochastic asymptotics. We also present experiments from regression, classification, and off-policy prediction to demonstrate that this is a general phenomenon. We argue that it is imperative to consider the objective function design and the optimization algorithm together while addressing the sampling bias.