How does overparametrization affect performance on minority groups?
Not on the map. Why?
Abstract (arXiv)
The benefits of overparameterization for the overall performance of modern machine learning (ML) models are well known. However, the effect of overparameterization at a more granular level of data subgroups is less understood. Recent empirical studies demonstrate encouraging results: (i) when groups are not known, overparameterized models trained with empirical risk minimization (ERM) perform better on minority groups; (ii) when groups are known, ERM on data subsampled to equalize group sizes yields state-of-the-art worst-group-accuracy in the overparameterized regime. In this paper, we complement these empirical studies with a theoretical investigation of the risk of overparameterized random feature models on minority groups. In a setting in which the regression functions for the majority and minority groups are different, we show that overparameterization always improves minority group performance.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 7 Jun 2022.
Builds on
25 of 41 references.
References · 25
- 2016Equality of Opportunity in Supervised Learningcited by 2,000
- 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
- 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2018A Reductions Approach to Fair Classificationcited by 585
- 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
- 2020Performative Predictioncited by 318
- 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
- 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
13 more
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
- 2019Two models of double descent for weak featurescited by 194
- 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
- 2018The Frontiers of Fairness in Machine Learningcited by 139
- 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
- 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
- 2019Predictive Inequity in Object Detectioncited by 77
- 2022Fair Contrastive Learning for Facial Attribute Classificationcited by 71
- 2020Distributionally Robust Losses for Latent Covariate Mixturescited by 57
- 2021DORO: Distributional and Outlier Robust Optimizationcited by 54
- 2021Towards Fair Classifiers Without Sensitive Attributes: Exploring Biases in Related Featurescited by 45
- 2020Why resampling outperforms reweighting for correcting sampling bias with stochastic gradientscited by 34
- 2021The Effect of Model Size on Worst-Group Generalizationcited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 1.
2 papers.