mlsanityData to 7 Oct 2026

Rethinking the Evaluation Protocol of Domain Generalization

arXiv 2305.15253 ↗First posted 24 May 2023cs.LG cs.AI cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Domain generalization aims to solve the challenge of Out-of-Distribution (OOD) generalization by leveraging common knowledge learned from multiple training domains to generalize to unseen test domains. To accurately evaluate the OOD generalization ability, it is required that test data information is unavailable. However, the current domain generalization protocol may still have potential test data information leakage. This paper examines the risks of test data information leakage from two aspects of the current evaluation protocol: supervised pretraining on ImageNet and oracle model selection. We propose modifications to the current protocol that we should employ self-supervised pretraining or train from scratch instead of employing the current supervised pretraining, and we should use multiple test domains. These would result in a more precise evaluation of OOD generalization ability. We also rerun the algorithms with the modified protocol and introduce new leaderboards to encourage future research in domain generalization with a fairer comparison.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 72 of its 100 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

72 of 100 references.

References · 72

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2019Invariant Risk Minimizationcited by 1,832
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  3. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  6. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  9. 2020In Search of Lost Domain Generalizationcited by 1,042
  10. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  11. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  12. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  13. 2021Domain Generalization: A Surveycited by 760
  14. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  15. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  16. 2018Recognition in Terra Incognitacited by 655
  17. 2021Domain Generalization with MixStylecited by 563
  18. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  19. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  20. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  21. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 449
  22. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  23. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  24. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  25. 2015Return of Frustratingly Easy Domain Adaptationcited by 389
  26. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  27. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  28. 2017Fairness in Criminal Justice Risk Assessments: The State of the Artcited by 315
  29. 2020Environment Inference for Invariant Learningcited by 306
  30. 2018Generalizing Across Domains via Cross-Gradient Trainingcited by 268
  31. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  32. 2019Episodic Training for Domain Generalizationcited by 245
  33. 2020Domain Generalization using Causal Matchingcited by 224
  34. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  35. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  36. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  37. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  38. 2019Domain Generalization Using a Mixture of Multiple Latent Domainscited by 170

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  2. 2024Is Large-Scale Pretraining the Secret to Good Domain Generalization?cited by 9
  3. 2025Error Slice Discovery via Manifold Compactnesscited by 4
  4. 2025Estimating Control Barriers from Offline Datacited by 3
  5. 2025Measure-Theoretic Anti-Causal Representation Learningcited by 3
  6. 2024GAQAT: gradient-adaptive quantization-aware training for domain generalizationcited by 2
  7. 2024Temporal Variability and Multi-Viewed Self-Supervised Representations to Tackle the ASVspoof5 Deepfake Challengecited by 1
  8. 2024R+R: Revisiting Static Feature-Based Android Malware Detection using Machine Learningcited by 1
  9. 2025COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shiftscited by 1
  10. 2025Flatness-aware Curriculum Learning via Adversarial Difficultycited by 1
  11. 2025ConstStyle: Robust Domain Generalization with Unified Style Transformationcited by 1
  12. 2025ODP-Bench: Benchmarking Out-of-Distribution Performance Predictioncited by 1
8 more
  1. 2025FixCLR: Negative-Class Contrastive Learning for Semi-Supervised Domain Generalizationcited by 0
  2. 2025Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representationscited by 0
  3. 2025Generating Risky Samples with Conformity Constraints via Diffusion Modelscited by 0
  4. 2026WildIng: A Wildlife Image Invariant Representation Model for Geographical Domain Shiftcited by 0
  5. 2026Toward Multi-Domain and Long-Tailed Quantization via Feature Alignment and Scalingcited by 0
  6. 2026Domain Generalization via Text-Anchored Information Bottleneckcited by 0
  7. 2026Acoustic domain shift in spoken language identification from systematic domain generalization evaluation to real-world applicationcited by 0
  8. 2026Advancing Wildlife Conservation through Multimodal Animal Re-Identification with Environmental Metadatacited by 0