mlsanityData to 7 Oct 2026

The Effect of Model Size on Worst-Group Generalization

arXiv 2112.04094 ↗First posted 8 Dec 2021cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Overparameterization is shown to result in poor test accuracy on rare subgroups under a variety of settings where subgroup information is known. To gain a more complete picture, we consider the case where subgroup information is unknown. We investigate the effect of model size on worst-group generalization under empirical risk minimization (ERM) across a wide range of settings, varying: 1) architectures (ResNet, VGG, or BERT), 2) domains (vision or natural language processing), 3) model size (width or depth), and 4) initialization (with pre-trained or random weights). Our systematic evaluation reveals that increasing model size does not hurt, and may help, worst-group test performance under ERM across all setups. In particular, increasing pre-trained model size consistently improves performance on Waterbirds and MultiNLI. We advise practitioners to use larger pre-trained models when subgroup labels are unknown.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

27 of 36 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  11. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  12. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
15 more
  1. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  2. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  3. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  4. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  5. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  6. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  7. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  8. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  9. 2020Model Patching: Closing the Subgroup Performance Gap with Data Augmentationcited by 82
  10. 2021Causal Attention for Unbiased Visual Recognitioncited by 80
  11. 2021Examining and Combating Spurious Features under Distribution Shiftcited by 64
  12. 2020Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionatelycited by 52
  13. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  14. 2020Small Data, Big Decisions: Model Selection in the Small-Data Regimecited by 23
  15. 2019HBONet: Harmonious Bottleneck on Two Orthogonal Dimensionscited by 13

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1; 2025: 1.

4 papers.

  1. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
  2. 2024The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlationscited by 5
  3. 2022Fairness via In-Processing in the Over-parameterized Regime: A Cautionary Talecited by 4
  4. 2022How does overparametrization affect performance on minority groups?cited by 2