Revisiting the Dataset Bias Problem from a Statistical Perspective
Not on the map. Why?
Abstract (arXiv)
In this paper, we study the "dataset bias" problem from a statistical standpoint, and identify the main cause of the problem as the strong correlation between a class attribute u and a non-class attribute b in the input x, represented by p(u|b) differing significantly from p(u). Since p(u|b) appears as part of the sampling distributions in the standard maximum log-likelihood (MLL) objective, a model trained on a biased dataset via MLL inherently incorporates such correlation into its parameters, leading to poor generalization to unbiased test data. From this observation, we propose to mitigate dataset bias via either weighting the objective of each sample n by frac1p(un|bn) or sampling that sample with a weight proportional to frac1p(un|bn). While both methods are statistically equivalent, the former proves more stable and effective in practice. Additionally, we establish a connection between our debiasing approach and causal reasoning, reinforcing our method's theoretical foundation. However, when the bias label is unavailable, computing p(u|b) exactly is difficult. To overcome this challenge, we propose to approximate frac1p(u|b) using a biased classifier trained with "bias amplification" losses. Extensive experiments on various biased datasets demonstrate the superiority of our method over existing debiasing techniques in most settings, validating our theoretical analysis.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 65 references that are in the map. First posted on arXiv 5 Feb 2024.
Builds on
40 of 65 references.
References · 40
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2017Neural Discrete Representation Learningcited by 4,650
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2015Domain-Adversarial Training of Neural Networkscited by 3,023
- 2015Learning Deep Features for Discriminative Localizationcited by 2,430
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
- 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
- 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
- 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
- 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
- 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
28 more
- 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
- 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
- 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
- 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
- 2020A Survey on Causal Inferencecited by 202
- 2019Learning De-biased Representations with Biased Representationscited by 199
- 2020Representation Learning via Invariant Causal Mechanismscited by 197
- 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
- 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
- 2015Cost Sensitive Learning of Deep Feature Representations from Imbalanced Datacited by 167
- 2020Adversarial Filters of Dataset Biasescited by 159
- 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
- 2016Discrete Variational Autoencoderscited by 124
- 2019Learning Robust Representations by Projecting Superficial Statistics Outcited by 123
- 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
- 2021Learning Debiased Representation via Disentangled Feature Augmentationcited by 116
- 2018Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddingscited by 114
- 2021EnD: Entangling and Disentangling deep representations for bias correctioncited by 105
- 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
- 2019Striking the Right Balance with Uncertaintycited by 79
- 2021BiaSwap: Removing dataset bias with bias-tailored swapping augmentationcited by 70
- 2022Learning Debiased Classifier with Biased Committeecited by 51
- 2022Out-of-distribution Generalization with Causal Invariant Transformationscited by 45
- 2021Transporting Causal Mechanisms for Unsupervised Domain Adaptationcited by 38
- 2021Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimizationcited by 32
- 2022SelecMix: Debiased Learning by Contradicting-pair Samplingcited by 29
- 2022Mitigating Dataset Bias by Using Per-sample Gradientcited by 21
- 2021Are Bias Mitigation Techniques for Deep Learning Effective?cited by 14
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Reducing Class Bias In Data-Balanced Datasets Through Hardness-Based Resampling
- 2024Training Unbiased Diffusion Models From Biased Dataset
- 2026The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
- 2024DiffInject: Revisiting Debias via Synthetic Data Generation using Diffusion-based Style Injection
- 2024The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes
- 2024Do we need rebalancing strategies? A theoretical and empirical study around SMOTE and its variants
- 2024Rethinking Fair Representation Learning for Performance-Sensitive Tasks
- 2024The Limits of Inference Scaling Through Resampling