# A Survey on Evaluation of Out-of-Distribution Generalization

arXiv: https://arxiv.org/abs/2403.01874 · HTML: https://mlsanity.com/paper/2403.01874/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 113 of its 148 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 113)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)

## Built on by (most cited first, 21 of 21)

- [Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Control](https://mlsanity.com/paper/2509.15130/) (2025)
- [VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models](https://mlsanity.com/paper/2603.06148/) (2026)
- [Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox](https://mlsanity.com/paper/2406.12928/) (2024)
- [Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Report](https://mlsanity.com/paper/2507.06968/) (2025)
- [Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics](https://mlsanity.com/paper/2406.05108/) (2024)
- [Error Slice Discovery via Manifold Compactness](https://mlsanity.com/paper/2501.19032/) (2025)
- [GAQAT: gradient-adaptive quantization-aware training for domain generalization](https://mlsanity.com/paper/2412.05551/) (2024)
- [Model Assembly Learning with Heterogeneous Layer Weight Merging](https://mlsanity.com/paper/2503.21657/) (2025)
- [Boosting Test Performance with Importance Sampling--a Subpopulation Perspective](https://mlsanity.com/paper/2412.13003/) (2024)
- [Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks](https://mlsanity.com/paper/2412.20682/) (2024)
- [BroadGen: A Framework for Generating Effective and Efficient Advertiser Broad Match Keyphrase Recommendations](https://mlsanity.com/paper/2505.19164/) (2025)
- [ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction](https://mlsanity.com/paper/2510.27263/) (2025)
- [Testing Generalizability in Causal Inference](https://mlsanity.com/paper/2411.03021/) (2024)
- [CRTRE: Causal Rule Generation with Target Trial Emulation Framework](https://mlsanity.com/paper/2411.06338/) (2024)
- [SAMOSA: Sharpness Aware Minimization for Open Set Active learning](https://mlsanity.com/paper/2510.16757/) (2025)
- [Contrast transfer functions help quantify neural network out-of-distribution generalization in HRTEM](https://mlsanity.com/paper/2512.09067/) (2025)
- [Semantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content](https://mlsanity.com/paper/2601.05103/) (2026)
- [Do Transformers Have the Ability for Periodicity Generalization?](https://mlsanity.com/paper/2601.22690/) (2026)
- [Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings](https://mlsanity.com/paper/2604.08192/) (2026)
- [The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms](https://mlsanity.com/paper/2606.25450/) (2026)
- [Learning the Robustness Mechanism with Bilevel Optimization](https://mlsanity.com/paper/2609.35541/) (2026)

## Abstract (arXiv)

Machine learning models, while progressively advanced, rely heavily on the IID assumption, which is often unfulfilled in practice due to inevitable distribution shifts. This renders them susceptible and untrustworthy for deployment in risk-sensitive applications. Such a significant problem has consequently spawned various branches of works dedicated to developing algorithms capable of Out-of-Distribution (OOD) generalization. Despite these efforts, much less attention has been paid to the evaluation of OOD generalization, which is also a complex and fundamental problem. Its goal is not only to assess whether a model's OOD generalization capability is strong or not, but also to evaluate where a model generalizes well or poorly. This entails characterizing the types of distribution shifts that a model can effectively address, and identifying the safe and risky input regions given a model. This paper serves as the first effort to conduct a comprehensive review of OOD evaluation. We categorize existing research into three paradigms: OOD performance testing, OOD performance prediction, and OOD intrinsic property characterization, according to the availability of test data. Additionally, we briefly discuss OOD evaluation in the context of pretrained models. In closing, we propose several promising directions for future research in OOD evaluation.