# The Effect of Model Size on Worst-Group Generalization

arXiv: https://arxiv.org/abs/2112.04094 · HTML: https://mlsanity.com/paper/2112.04094/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [Rethinking Bias-Variance Trade-off for Generalization of Neural Networks](https://mlsanity.com/paper/2002.11328/) (2020)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [Causal Attention for Unbiased Visual Recognition](https://mlsanity.com/paper/2108.08782/) (2021)
- [Examining and Combating Spurious Features under Distribution Shift](https://mlsanity.com/paper/2106.07171/) (2021)
- [Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately](https://mlsanity.com/paper/2012.04104/) (2020)
- [Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level](https://mlsanity.com/paper/2105.06020/) (2021)
- [Small Data, Big Decisions: Model Selection in the Small-Data Regime](https://mlsanity.com/paper/2009.12583/) (2020)
- [HBONet: Harmonious Bottleneck on Two Orthogonal Dimensions](https://mlsanity.com/paper/1908.03888/) (2019)

## Built on by (most cited first, 4 of 4)

- [Steering CLIP's vision transformer with sparse autoencoders](https://mlsanity.com/paper/2504.08729/) (2025)
- [The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations](https://mlsanity.com/paper/2407.13957/) (2024)
- [Fairness via In-Processing in the Over-parameterized Regime: A Cautionary Tale](https://mlsanity.com/paper/2206.14853/) (2022)
- [How does overparametrization affect performance on minority groups?](https://mlsanity.com/paper/2206.03515/) (2022)

## Abstract (arXiv)

Overparameterization is shown to result in poor test accuracy on rare subgroups under a variety of settings where subgroup information is known. To gain a more complete picture, we consider the case where subgroup information is unknown. We investigate the effect of model size on worst-group generalization under empirical risk minimization (ERM) across a wide range of settings, varying: 1) architectures (ResNet, VGG, or BERT), 2) domains (vision or natural language processing), 3) model size (width or depth), and 4) initialization (with pre-trained or random weights). Our systematic evaluation reveals that increasing model size does not hurt, and may help, worst-group test performance under ERM across all setups. In particular, increasing pre-trained model size consistently improves performance on Waterbirds and MultiNLI. We advise practitioners to use larger pre-trained models when subgroup labels are unknown.