# Linear unit-tests for invariance discovery

arXiv: https://arxiv.org/abs/2102.10867 · HTML: https://mlsanity.com/paper/2102.10867/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Learning explanations that are hard to vary](https://mlsanity.com/paper/2009.00329/) (2020)

## Built on by (most cited first, 27 of 27)

- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2202.05441/) (2022)
- [Provable Domain Generalization via Invariant-Feature Subspace Recovery](https://mlsanity.com/paper/2201.12919/) (2022)
- [Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments](https://mlsanity.com/paper/2106.09913/) (2021)
- [WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series](https://mlsanity.com/paper/2203.09978/) (2022)
- [Explicit Tradeoffs between Adversarial and Natural Distributional Robustness](https://mlsanity.com/paper/2209.07592/) (2022)
- ["Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts](https://mlsanity.com/paper/2210.10769/) (2022)
- [VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution](https://mlsanity.com/paper/2304.01434/) (2023)
- [Towards Robust Out-of-Distribution Generalization Bounds via Sharpness](https://mlsanity.com/paper/2403.06392/) (2024)
- [On Invariance Penalties for Risk Minimization](https://mlsanity.com/paper/2106.09777/) (2021)
- [Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles](https://mlsanity.com/paper/2109.10432/) (2021)
- [Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift](https://mlsanity.com/paper/2310.04971/) (2023)
- [Conditional entropy minimization principle for learning domain invariant representation features](https://mlsanity.com/paper/2201.10460/) (2022)
- [Decorr: Environment Partitioning for Invariant Learning and OOD Generalization](https://mlsanity.com/paper/2211.10054/) (2022)
- [Few-shot Adaptation to Distribution Shifts By Mixing Source and Target Embeddings](https://mlsanity.com/paper/2305.14521/) (2023)
- [Identifying and Benchmarking Natural Out-of-Context Prediction Problems](https://mlsanity.com/paper/2110.13223/) (2021)
- [Trade-off between reconstruction loss and feature alignment for domain generalization](https://mlsanity.com/paper/2210.15000/) (2022)
- [Invariant-Feature Subspace Recovery: A New Class of Provable Domain Generalization Algorithms](https://mlsanity.com/paper/2311.00966/) (2023)
- [A call for better unit testing for invariant risk minimisation](https://mlsanity.com/paper/2106.03234/) (2021)
- [Invariant Risk Minimisation for Cross-Organism Inference: Substituting Mouse Data for Human Data in Human Risk Factor Discovery](https://mlsanity.com/paper/2111.07348/) (2021)
- [Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2208.07798/) (2022)
- [Spuriosity Rankings for Free: A Simple Framework for Last Layer Retraining Based on Object Detection](https://mlsanity.com/paper/2311.00079/) (2023)
- [Towards a Better Evaluation of Out-of-Domain Generalization](https://mlsanity.com/paper/2405.19703/) (2024)
- [Feature Matching Intervention: Leveraging Observational Data for Causal Representation Learning](https://mlsanity.com/paper/2503.03634/) (2025)
- [PISA: Prioritized Invariant Subgraph Aggregation for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2511.22435/) (2025)

## Abstract (arXiv)

There is an increasing interest in algorithms to learn invariant correlations across training environments. A big share of the current proposals find theoretical support in the causality literature but, how useful are they in practice? The purpose of this note is to propose six linear low-dimensional problems -- unit tests -- to evaluate different types of out-of-distribution generalization in a precise manner. Following initial experiments, none of the three recently proposed alternatives passes all tests. By providing the code to automatically replicate all the results in this manuscript (https://www.github.com/facebookresearch/InvarianceUnitTests), we hope that our unit tests become a standard steppingstone for researchers in out-of-distribution generalization.