mlsanityData to 7 Oct 2026

Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing

arXiv 2410.05980 ↗First posted 8 Oct 2024cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

As training datasets grow larger, we aspire to develop models that generalize well to any diverse test distribution, even if the latter deviates significantly from the training data. Various approaches like domain adaptation, domain generalization, and robust optimization attempt to address the out-of-distribution challenge by posing assumptions about the relation between training and test distribution. Differently, we adopt a more conservative perspective by accounting for the worst-case error across all sufficiently diverse test distributions within a known domain. Our first finding is that training on a uniform distribution over this domain is optimal. We also interrogate practical remedies when uniform samples are unavailable by considering methods for mitigating non-uniformity through finetuning and rebalancing. Our theory provides a mathematical grounding for previous observations on the role of entropy and rebalancing for o.o.d. generalization and foundation model training. We also provide new empirical evidence across tasks involving o.o.d. shifts which illustrate the broad applicability of our perspective.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

38 of 63 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2019Invariant Risk Minimizationcited by 1,832
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2020In Search of Lost Domain Generalizationcited by 1,042
  12. 2017Deep Bayesian Active Learning with Image Datacited by 812
26 more
  1. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  2. 2021Domain Generalization: A Surveycited by 760
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  5. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  6. 2020A Brief Review of Domain Adaptationcited by 287
  7. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  8. 2020The Risks of Invariant Risk Minimizationcited by 262
  9. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  10. 2015Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understandingcited by 218
  11. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  12. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  13. 2019Are Labels Required for Improving Adversarial Robustness?cited by 162
  14. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  15. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  16. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  17. 2021Assessing Generalization of SGD via Disagreementcited by 116
  18. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  19. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  20. 2021The iWildCam 2021 Competition Datasetcited by 77
  21. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  22. 2020When is invariance useful in an Out-of-Distribution Generalization problem ?cited by 52
  23. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  24. 2021Measuring Generalization with Optimal Transportcited by 24
  25. 2022Why does Throwing Away Data Improve Worst-Group Error?cited by 22
  26. 2022Predicting Out-of-Domain Generalization with Neighborhood Invariancecited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Conformation-Aware Structure Prediction of Antigen-Recognizing Immune Proteinscited by 2
  2. 2025Composable and adaptive design of machine learning interatomic potentials guided by Fisher-information analysiscited by 0