# On Invariance Penalties for Risk Minimization

arXiv: https://arxiv.org/abs/2106.09777 · HTML: https://mlsanity.com/paper/2106.09777/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 13 of 13)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)
- [Learning explanations that are hard to vary](https://mlsanity.com/paper/2009.00329/) (2020)
- [Does Invariant Risk Minimization Capture Invariance?](https://mlsanity.com/paper/2101.01134/) (2021)
- [Out of Distribution Generalization in Machine Learning](https://mlsanity.com/paper/2103.02667/) (2021)
- [Unshuffling Data for Improved Generalization](https://mlsanity.com/paper/2002.11894/) (2020)
- [Linear unit-tests for invariance discovery](https://mlsanity.com/paper/2102.10867/) (2021)
- [Invariant Representation Learning for Treatment Effect Estimation](https://mlsanity.com/paper/2011.12379/) (2020)
- [Learning Causal Models Online](https://mlsanity.com/paper/2006.07461/) (2020)

## Built on by (most cited first, 10 of 10)

- [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [C-Mixup: Improving Generalization in Regression](https://mlsanity.com/paper/2210.05775/) (2022)
- [Provable Domain Generalization via Invariant-Feature Subspace Recovery](https://mlsanity.com/paper/2201.12919/) (2022)
- [Learning Invariant Visual Representations for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2206.00415/) (2022)
- [Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noise](https://mlsanity.com/paper/2210.11075/) (2022)
- [Improving Domain Generalization with Domain Relations](https://mlsanity.com/paper/2302.02609/) (2023)
- [Multi-Domain Long-Tailed Learning by Augmenting Disentangled Representations](https://mlsanity.com/paper/2210.14358/) (2022)
- [Invariant-Feature Subspace Recovery: A New Class of Provable Domain Generalization Algorithms](https://mlsanity.com/paper/2311.00966/) (2023)
- [Global Layers: Non-IID Tabular Federated Learning](https://mlsanity.com/paper/2305.19290/) (2023)

## Abstract (arXiv)

The Invariant Risk Minimization (IRM) principle was first proposed by Arjovsky et al. [2019] to address the domain generalization problem by leveraging data heterogeneity from differing experimental conditions. Specifically, IRM seeks to find a data representation under which an optimal classifier remains invariant across all domains. Despite the conceptual appeal of IRM, the effectiveness of the originally proposed invariance penalty has recently been brought into question. In particular, there exists counterexamples for which that invariance penalty can be arbitrarily small for non-invariant data representations. We propose an alternative invariance penalty by revisiting the Gramian matrix of the data representation. We discuss the role of its eigenvalues in the relationship between the risk and the invariance penalty, and demonstrate that it is ill-conditioned for said counterexamples. The proposed approach is guaranteed to recover an invariant representation for linear settings under mild non-degeneracy conditions. Its effectiveness is substantiated by experiments on DomainBed and InvarianceUnitTest, two extensive test beds for domain generalization.