mlsanityData to 7 Oct 2026

On how to avoid exacerbating spurious correlations when models are overparameterized

arXiv 2206.12739 ↗First posted 25 Jun 2022cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Overparameterized models fail to generalize well in the presence of data imbalance even when combined with traditional techniques for mitigating imbalances. This paper focuses on imbalanced classification datasets, in which a small subset of the population -- a minority -- may contain features that correlate spuriously with the class label. For a parametric family of cross-entropy loss modifications and a representative Gaussian mixture model, we derive non-asymptotic generalization bounds on the worst-group error that shed light on the role of different hyper-parameters. Specifically, we prove that, when appropriately tuned, the recently proposed VS-loss learns a model that is fair towards minorities even when spurious features are strong. On the other hand, alternative heuristics, such as the weighted CE and the LA-loss, can fail dramatically. Compared to previous works, our bounds hold for more general models, they are non-asymptotic, and, they apply even at scenarios of extreme imbalance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 25 Jun 2022.

Builds on

33 of 39 references.

References · 33

  1. 2019Invariant Risk Minimizationcited by 1,832
  2. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  3. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2020Long-tail learning via logit adjustmentcited by 605
  8. 2019Benign Overfitting in Linear Regressioncited by 544
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  10. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  11. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  12. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
21 more
  1. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  2. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  3. 2018Risk and parameter convergence of logistic regressioncited by 81
  4. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  5. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  6. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  7. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  8. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  9. 2020Gradient descent follows the regularization path for general lossescited by 57
  10. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  11. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  12. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  13. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  14. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  15. 2013A framework to characterize performance of LASSO algorithmscited by 38
  16. 2020On the proliferation of support vectors in high dimensionscited by 32
  17. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  18. 2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
  19. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18
  20. 2020Sharp Asymptotics and Optimal Performance for Inference in Binary Modelscited by 13
  21. 2021Support vector machines and linear regression coincide with very high-dimensional featurescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 1.

4 papers.

  1. 2023General Loss Functions Lead to (Approximate) Interpolation in High Dimensionscited by 4
  2. 2024Sharp analysis of out-of-distribution error for "importance-weighted" estimators in the overparameterized regimecited by 3
  3. 2025Thumb on the Scale: Optimal Loss Weighting in Last Layer Retrainingcited by 1
  4. 2026Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Featurescited by 1