# Distribution Shift Is Key to Learning Invariant Prediction

arXiv: https://arxiv.org/abs/2601.12296 · HTML: https://mlsanity.com/paper/2601.12296/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 18 Jan 2026.

## Builds on (most cited first, 18 of 18)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Domain Generalization via Conditional Invariant Representation](https://mlsanity.com/paper/1807.08479/) (2018)
- [Invariance, Causality and Robustness](https://mlsanity.com/paper/1812.08233/) (2018)
- [ZIN: When and How to Learn Invariance Without Environment Partition?](https://mlsanity.com/paper/2203.05818/) (2022)
- [Estimating Generalization under Distribution Shifts via Domain-Invariant Representations](https://mlsanity.com/paper/2007.03511/) (2020)
- [Fundamental Limits and Tradeoffs in Invariant Representation Learning](https://mlsanity.com/paper/2012.10713/) (2020)
- [Provable Domain Generalization via Invariant-Feature Subspace Recovery](https://mlsanity.com/paper/2201.12919/) (2022)
- [Causal Representation Learning Made Identifiable by Grouping of Observational Variables](https://mlsanity.com/paper/2310.15709/) (2023)
- [ERM++: An Improved Baseline for Domain Generalization](https://mlsanity.com/paper/2304.01973/) (2023)
- [Domain Generalisation via Risk Distribution Matching](https://mlsanity.com/paper/2310.18598/) (2023)
- [Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting](https://mlsanity.com/paper/2302.08635/) (2023)
- [The Missing Invariance Principle Found -- the Reciprocal Twin of Invariant Risk Minimization](https://mlsanity.com/paper/2205.14546/) (2022)
- [Learning Optimal Features via Partial Invariance](https://mlsanity.com/paper/2301.12067/) (2023)

## Built on by (most cited first, 2 of 2)

- [How Many Samples Are Enough for Learning Across Domains?](https://mlsanity.com/paper/2609.39336/) (2026)
- [Can Domain Generalization be Guaranteed in Small-Sample Learning?](https://mlsanity.com/paper/2609.39512/) (2026)

## Abstract (arXiv)

An interesting phenomenon arises: Empirical Risk Minimization (ERM) sometimes outperforms methods specifically designed for out-of-distribution tasks. This motivates an investigation into the reasons behind such behavior beyond algorithmic design. In this study, we find that one such reason lies in the distribution shift across training domains. A large degree of distribution shift can lead to better performance even under ERM. Specifically, we derive several theoretical and empirical findings demonstrating that distribution shift plays a crucial role in model learning and benefits learning invariant prediction. Firstly, the proposed upper bounds indicate that the degree of distribution shift directly affects the prediction ability of the learned models. If it is large, the models' ability can increase, approximating invariant prediction models that make stable predictions under arbitrary known or unseen domains; and vice versa. We also prove that, under certain data conditions, ERM solutions can achieve performance comparable to that of invariant prediction models. Secondly, the empirical validation results demonstrated that the predictions of learned models approximate those of Oracle or Optimal models, provided that the degree of distribution shift in the training data increases.