# Out-of-Distribution Generalization via Risk Extrapolation (REx)

arXiv: https://arxiv.org/abs/2003.00688 · HTML: https://mlsanity.com/paper/2003.00688/

## Key facts

- Cited by 742 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 30 of 742)

- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022)
- [SelfReg: Self-supervised Contrastive Regularization for Domain Generalization](https://mlsanity.com/paper/2104.09841/) (2021)
- [Discovering Invariant Rationales for Graph Neural Networks](https://mlsanity.com/paper/2201.12872/) (2022)
- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations](https://mlsanity.com/paper/2203.01517/) (2022)
- [Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism](https://mlsanity.com/paper/2201.12987/) (2022)
- [Handling Distribution Shifts on Graphs: An Invariance Perspective](https://mlsanity.com/paper/2202.02466/) (2022)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization](https://mlsanity.com/paper/2110.10832/) (2021)
- [Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2202.05441/) (2022)
- [GOOD: A Graph Out-of-Distribution Benchmark](https://mlsanity.com/paper/2206.08452/) (2022)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Domain Generalization by Mutual-Information Regularization with Pre-trained Models](https://mlsanity.com/paper/2203.10789/) (2022)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Model-Based Domain Generalization](https://mlsanity.com/paper/2102.11436/) (2021)
- [Sharpness-Aware Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2303.10353/) (2023)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022)
- [Towards a Theoretical Framework of Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.04496/) (2021)
- [Towards Principled Disentanglement for Domain Generalization](https://mlsanity.com/paper/2111.13839/) (2021)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)

## Abstract (arXiv)

Distributional shift is one of the major obstacles when transferring machine learning prediction systems from the lab to the real world. To tackle this problem, we assume that variation across training domains is representative of the variation we might encounter at test time, but also that shifts at test time may be more extreme in magnitude. In particular, we show that reducing differences in risk across training domains can reduce a model's sensitivity to a wide range of extreme distributional shifts, including the challenging setting where the input contains both causal and anti-causal elements. We motivate this approach, Risk Extrapolation (REx), as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and propose a penalty on the variance of training risks (V-REx) as a simpler variant. We prove that variants of REx can recover the causal mechanisms of the targets, while also providing some robustness to changes in the input distribution ("covariate shift"). By appropriately trading-off robustness to causally induced distributional shifts and covariate shift, REx is able to outperform alternative methods such as Invariant Risk Minimization in situations where these types of shift co-occur.