mlsanityData to 14 Oct 2026

Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization

arXiv 1911.08731 ↗First posted 20 Nov 2019cs.LG stat.MLcited by 1,247

Not on the map. Why?

Cited by 42 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Overparameterized neural networks can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such groups). Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. However, we find that naively applying group DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss also already has vanishing worst-case training loss. Instead, the poor worst-case performance arises from poor generalization on some groups. By coupling group DRO models with increased regularization---a stronger-than-typical L2 penalty or early stopping---we achieve substantially higher worst-group accuracies, with 10-40 percentage point improvements on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is important for worst-group generalization in the overparameterized regime, even if it is not needed for average generalization. Finally, we introduce a stochastic optimization algorithm, with convergence guarantees, to efficiently train group DRO models.

Key facts. Cited by 1,247 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 113; 2022: 207; 2023: 236; 2024: 239; 2025: 205; 2026: 247.

1,247 papers.

Sort:

By area:

  1. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  2. 2021Domain Generalization: A Surveycited by 764
  3. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  5. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  7. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  8. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377
  9. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  10. 2021Gradient Matching for Domain Generalizationcited by 258
  11. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  12. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
38 more
  1. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  2. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  3. 2021SelfReg: Self-supervised Contrastive Regularization for Domain Generalizationcited by 189
  4. 2022Discovering Invariant Rationales for Graph Neural Networkscited by 188
  5. 2021Simple data balancing achieves competitive worst-group-accuracycited by 178
  6. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  7. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  8. 2021Meta-learning via Language Model In-context Tuningcited by 164
  9. 2022Handling Distribution Shifts on Graphs: An Invariance Perspectivecited by 164
  10. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  11. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  12. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  13. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  14. 2022GOOD: A Graph Out-of-Distribution Benchmarkcited by 141
  15. 2022Causality Inspired Representation Learning for Domain Generalizationcited by 138
  16. 2021On Calibration and Out-of-domain Generalizationcited by 134
  17. 2021Disparities in Dermatology AI: Assessments Using Diverse Clinical Imagescited by 134
  18. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  19. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 120
  20. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 119
  21. 2021Learning Debiased Representation via Disentangled Feature Augmentationcited by 116
  22. 2021Causal Attention for Interpretable and Generalizable Graph Classificationcited by 108
  23. 2021Model-Based Domain Generalizationcited by 104
  24. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  25. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  26. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  27. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  28. 2023Eliciting Human Preferences with Language Modelscited by 92
  29. 2021Towards Principled Disentanglement for Domain Generalizationcited by 91
  30. 2021Heterogeneous Risk Minimizationcited by 90
  31. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
  32. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  33. 2022Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimationcited by 88
  34. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  35. 2021Exploiting Domain-Specific Features to Enhance Domain Generalizationcited by 86
  36. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 86
  37. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  38. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85

50 most cited of 1,247 All 1,247 (JSON)