# Provable Domain Generalization via Invariant-Feature Subspace Recovery

arXiv: https://arxiv.org/abs/2201.12919 · HTML: https://mlsanity.com/paper/2201.12919/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 80 references that are in the map.
- First posted on arXiv 30 Jan 2022.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)

## Built on by (most cited first, 30 of 32)

- [Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.06856/) (2022)
- [Out-of-Distribution Representation Learning for Time Series Classification](https://mlsanity.com/paper/2209.07027/) (2022)
- [Gradual Domain Adaptation: Theory and Algorithms](https://mlsanity.com/paper/2310.13852/) (2023)
- [A Review of the Role of Causality in Developing Trustworthy AI Systems](https://mlsanity.com/paper/2302.06975/) (2023)
- [Do causal predictors generalize better to new domains?](https://mlsanity.com/paper/2402.09891/) (2024)
- [Feature contamination: Neural networks learn uncorrelated features and fail to generalize](https://mlsanity.com/paper/2406.03345/) (2024)
- [Malign Overfitting: Interpolation Can Provably Preclude Invariance](https://mlsanity.com/paper/2211.15724/) (2022)
- [Enhancing Compositional Generalization via Compositional Feature Alignment](https://mlsanity.com/paper/2402.02851/) (2024)
- [Causally Inspired Regularization Enables Domain General Representations](https://mlsanity.com/paper/2404.16277/) (2024)
- [Future Gradient Descent for Adapting the Temporal Shifting Data Distribution in Online Recommendation Systems](https://mlsanity.com/paper/2209.01143/) (2022)
- [Target Conditioned Representation Independence (TCRI); From Domain-Invariant to Domain-General Representations](https://mlsanity.com/paper/2212.11342/) (2022)
- [Invariant Aggregator for Defending against Federated Backdoor Attacks](https://mlsanity.com/paper/2210.01834/) (2022)
- [Why Domain Generalization Fail? A View of Necessity and Sufficiency](https://mlsanity.com/paper/2502.10716/) (2025)
- [Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization](https://mlsanity.com/paper/2506.07378/) (2025)
- [Domain Generalization via Pareto Optimal Gradient Matching](https://mlsanity.com/paper/2507.14227/) (2025)
- [Distribution Shift Is Key to Learning Invariant Prediction](https://mlsanity.com/paper/2601.12296/) (2026)
- [PAC Generalization via Invariant Representations](https://mlsanity.com/paper/2205.15196/) (2022)
- [Constrained Maximum Cross-Domain Likelihood for Domain Generalization](https://mlsanity.com/paper/2210.04155/) (2022)
- [Context is Environment](https://mlsanity.com/paper/2309.09888/) (2023)
- [Robust Novelty Detection through Style-Conscious Feature Ranking](https://mlsanity.com/paper/2310.03738/) (2023)
- [Revisiting Spurious Correlation in Domain Generalization](https://mlsanity.com/paper/2406.11517/) (2024)
- [Human Heterogeneity Invariant Stress Sensing](https://mlsanity.com/paper/2506.02256/) (2025)
- [Anchor PCA](https://mlsanity.com/paper/2606.06233/) (2026)
- [Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2208.07798/) (2022)
- [Conformal Inference for Invariant Risk Minimization](https://mlsanity.com/paper/2305.12686/) (2023)
- [Environment-biased Feature Ranking for Novelty Detection Robustness](https://mlsanity.com/paper/2309.12301/) (2023)
- [SC2GAN: Rethinking Entanglement by Self-correcting Correlated GAN Space](https://mlsanity.com/paper/2310.06667/) (2023)
- [Preserving Silent Features for Domain Generalization](https://mlsanity.com/paper/2401.03170/) (2024)
- [A robust assessment for invariant representations](https://mlsanity.com/paper/2404.05058/) (2024)
- [BGM: Background Mixup for X-ray Prohibited Items Detection](https://mlsanity.com/paper/2412.00460/) (2024)

## Abstract (arXiv)

Domain generalization asks for models trained over a set of training environments to perform well in unseen test environments. Recently, a series of algorithms such as Invariant Risk Minimization (IRM) has been proposed for domain generalization. However, Rosenfeld et al. (2021) shows that in a simple linear data model, even if non-convexity issues are ignored, IRM and its extensions cannot generalize to unseen environments with less than ds+1 training environments, where ds is the dimension of the spurious-feature subspace. In this paper, we propose to achieve domain generalization with Invariant-feature Subspace Recovery (ISR). Our first algorithm, ISR-Mean, can identify the subspace spanned by invariant features from the first-order moments of the class-conditional distributions, and achieve provable domain generalization with ds+1 training environments under the data model of Rosenfeld et al. (2021). Our second algorithm, ISR-Cov, further reduces the required number of training environments to O(1) using the information of second-order moments. Notably, unlike IRM, our algorithms bypass non-convexity issues and enjoy global convergence guarantees. Empirically, our ISRs can obtain superior performance compared with IRM on synthetic benchmarks. In addition, on three real-world image and text datasets, we show that both ISRs can be used as simple yet effective post-processing methods to improve the worst-case accuracy of (pre-)trained models against spurious correlations and group shifts.