mlsanityData to 7 Oct 2026

Out-of-Distribution Generalization via Risk Extrapolation (REx)

arXiv 2003.00688 ↗First posted 2 Mar 2020cs.LG cs.AI cs.NEcited by 737

Not on the map. Why?

Abstract (arXiv)

Distributional shift is one of the major obstacles when transferring machine learning prediction systems from the lab to the real world. To tackle this problem, we assume that variation across training domains is representative of the variation we might encounter at test time, but also that shifts at test time may be more extreme in magnitude. In particular, we show that reducing differences in risk across training domains can reduce a model's sensitivity to a wide range of extreme distributional shifts, including the challenging setting where the input contains both causal and anti-causal elements. We motivate this approach, Risk Extrapolation (REx), as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and propose a penalty on the variance of training risks (V-REx) as a simpler variant. We prove that variants of REx can recover the causal mechanisms of the targets, while also providing some robustness to changes in the input distribution ("covariate shift"). By appropriately trading-off robustness to causally induced distributional shifts and covariate shift, REx is able to outperform alternative methods such as Invariant Risk Minimization in situations where these types of shift co-occur.

Key facts. Cited by 737 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 84; 2022: 152; 2023: 179; 2024: 121; 2025: 111; 2026: 90.

737 papers.

Sort:

By area:

  1. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  2. 2021Domain Generalization: A Surveycited by 760
  3. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  4. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  5. 2021Deep Long-Tailed Learning: A Surveycited by 416
  6. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  7. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  8. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  9. 2021SelfReg: Self-supervised Contrastive Regularization for Domain Generalizationcited by 189
  10. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  11. 2022Discovering Invariant Rationales for Graph Neural Networkscited by 186
  12. 2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
38 more
  1. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  2. 2022Handling Distribution Shifts on Graphs: An Invariance Perspectivecited by 163
  3. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  4. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  5. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  6. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  7. 2022GOOD: A Graph Out-of-Distribution Benchmarkcited by 140
  8. 2021On Calibration and Out-of-domain Generalizationcited by 134
  9. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  10. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  11. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  12. 2021Model-Based Domain Generalizationcited by 103
  13. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  14. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  15. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  16. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  17. 2021Towards Principled Disentanglement for Domain Generalizationcited by 91
  18. 2021OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalizationcited by 86
  19. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  20. 2021Exploiting Domain-Specific Features to Enhance Domain Generalizationcited by 86
  21. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  22. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  23. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  24. 2021Size-Invariant Graph Representations for Graph Classification Extrapolationscited by 82
  25. 2021Causal Attention for Unbiased Visual Recognitioncited by 80
  26. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  27. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  28. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  29. 2021Generalizing Graph Neural Networks on Out-Of-Distribution Graphscited by 69
  30. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  31. 2021On the Impact of Spurious Correlation for Out-of-distribution Detectioncited by 68
  32. 2022Invariant Grounding for Video Question Answeringcited by 68
  33. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  34. 2021MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaptioncited by 65
  35. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  36. 2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58
  37. 2022Balancing Discriminability and Transferability for Source-Free Domain Adaptationcited by 58
  38. 2022C-Mixup: Improving Generalization in Regressioncited by 57

50 most cited of 737 All 737 (JSON)