mlsanityData to 7 Oct 2026

Towards a Better Evaluation of Out-of-Domain Generalization

arXiv 2405.19703 ↗First posted 30 May 2024cs.LG cs.CV stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The objective of Domain Generalization (DG) is to devise algorithms and models capable of achieving high performance on previously unseen test distributions. In the pursuit of this objective, average measure has been employed as the prevalent measure for evaluating models and comparing algorithms in the existing DG studies. Despite its significance, a comprehensive exploration of the average measure has been lacking and its suitability in approximating the true domain generalization performance has been questionable. In this study, we carefully investigate the limitations inherent in the average measure and propose worst+gap measure as a robust alternative. We establish theoretical grounds of the proposed measure by deriving two theorems starting from two different assumptions. We conduct extensive experimental investigations to compare the proposed worst+gap measure with the conventional average measure. Given the indispensable need to access the true DG performance for studying measures, we modify five existing datasets to come up with SR-CMNIST, C-Cats&Dogs, L-CIFAR10, PACS-corrupted, and VLCS-corrupted datasets. The experiment results unveil an inferior performance of the average measure in approximating the true DG performance and confirm the robustness of the theoretically supported worst+gap measure.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

36 of 48 references.

References · 36

  1. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2019Invariant Risk Minimizationcited by 1,832
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  5. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  6. 2020In Search of Lost Domain Generalizationcited by 1,042
  7. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  8. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  9. 2021Domain Generalization: A Surveycited by 760
  10. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  11. 2013Domain Generalization via Invariant Feature Representationcited by 622
  12. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
24 more
  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  4. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  5. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  6. 2019Episodic Training for Domain Generalizationcited by 245
  7. 2020Domain Generalization using Causal Matchingcited by 224
  8. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  9. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  10. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  11. 2017Domain Generalization by Marginal Transfer Learningcited by 180
  12. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 160
  13. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  14. 2017Invariant Causal Prediction for Nonlinear Modelscited by 140
  15. 2020Heterogeneous Domain Generalization via Domain Mixupcited by 103
  16. 2021Model-Based Domain Generalizationcited by 103
  17. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  18. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  19. 2021Examining and Combating Spurious Features under Distribution Shiftcited by 64
  20. 2019Reducing Domain Gap by Reducing Style Biascited by 35
  21. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  22. 2021Linear unit-tests for invariance discoverycited by 27
  23. 2022Target Conditioned Representation Independence (TCRI); From Domain-Invariant to Domain-General Representationscited by 3
  24. 2022Adversarial and Random Transformations for Robust Domain Adaptation and Generalizationcited by 2

Built on by

Not cited yet.