mlsanityData to 7 Oct 2026

Towards Out-Of-Distribution Generalization: A Survey

arXiv 2108.13624 ↗First posted 31 Aug 2021cs.LGcited by 431

Not on the map. Why?

Abstract (arXiv)

Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed (i.i.d.). However, in real-world applications, this i.i.d. assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.

Key facts. Cited by 431 papers in the map (data through 7 Oct 2026). Builds on 196 of its 340 references that are in the map. First posted on arXiv 31 Aug 2021.

Builds on

196 of 340 references.

References · 196

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  10. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,350
38 more
  1. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  8. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2019Invariant Risk Minimizationcited by 1,832
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  15. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  16. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  17. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  18. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  19. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  20. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  21. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  22. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  23. 2018Sanity Checks for Saliency Mapscited by 1,149
  24. 2020In Search of Lost Domain Generalizationcited by 1,042
  25. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  26. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  27. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  28. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  29. 2017Counterfactual Fairnesscited by 802
  30. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  31. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  32. 2021Domain Generalization: A Surveycited by 760
  33. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  34. 2014Certifying and removing disparate impactcited by 715
  35. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  36. 2013Domain Generalization via Invariant Feature Representationcited by 622
  37. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  38. 2018Disentangling by Factorisingcited by 614

50 most cited of 196 All 196 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 78; 2023: 108; 2024: 104; 2025: 88; 2026: 45.

431 papers.

Sort:

By area:

  1. 2021Deep Long-Tailed Learning: A Surveycited by 416
  2. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  3. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  4. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  5. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  6. 2023All in One: Multi-task Prompting for Graph Neural Networkscited by 147
  7. 2022GOOD: A Graph Out-of-Distribution Benchmarkcited by 140
  8. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  9. 2022Uncertainty Modeling for Out-of-Distribution Generalizationcited by 122
  10. 2021Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspectivecited by 116
  11. 2022Out-Of-Distribution Generalization on Graphs: A Surveycited by 97
  12. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
38 more
  1. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  2. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  3. 2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 88
  4. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  5. 2024Robust agents learn causal world modelscited by 81
  6. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learningcited by 71
  7. 2023Compositional Generalization from First Principlescited by 62
  8. 2024Beyond Model Adaptation at Test Time: A Surveycited by 55
  9. 2024How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunitiescited by 50
  10. 2022Out-of-distribution generalization for learning quantum dynamicscited by 49
  11. 2023Domain Generalization for Medical Image Analysis: A Reviewcited by 47
  12. 2023Fusing Models with Complementary Expertisecited by 45
  13. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  14. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  15. 2023Neither hype nor gloom do DNNs justicecited by 40
  16. 2021Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domainscited by 39
  17. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  18. 2022Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shiftcited by 38
  19. 2023Learning to Augment Distributions for Out-of-Distribution Detectioncited by 38
  20. 2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Reviewcited by 37
  21. 2023Mind the Label Shift of Augmentation-based Graph OOD Generalizationcited by 34
  22. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  23. 2022Provable Domain Generalization via Invariant-Feature Subspace Recoverycited by 32
  24. 2024Alleviating Structural Distribution Shift in Graph Anomaly Detectioncited by 31
  25. 2024A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualizationcited by 31
  26. 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
  27. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  28. 2025Self-Improving LLM Agents at Test-Timecited by 30
  29. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  30. 2023Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalizationcited by 29
  31. 2024Transfer CLIP for Generalizable Image Denoisingcited by 29
  32. 2024Domain Generalization through Meta-Learning: A Surveycited by 28
  33. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  34. 2023Environment-Aware Dynamic Graph Learning for Out-of-Distribution Generalizationcited by 26
  35. 2024Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learningcited by 26
  36. 2023Federated Domain Generalization: A Surveycited by 25
  37. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  38. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24

50 most cited of 431 All 431 (JSON)