# Heterogeneous Risk Minimization

arXiv: https://arxiv.org/abs/2105.03818 · HTML: https://mlsanity.com/paper/2105.03818/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 9 May 2021.

## Builds on (most cited first, 13 of 13)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations](https://mlsanity.com/paper/1505.05116/) (2015)
- [Task2Vec: Task Embedding for Meta-Learning](https://mlsanity.com/paper/1902.03545/) (2019)
- [Invariant Rationalization](https://mlsanity.com/paper/2003.09772/) (2020)
- [Stable Prediction with Model Misspecification and Agnostic Distribution Shift](https://mlsanity.com/paper/2001.11713/) (2020)
- [Causally Regularized Learning with Agnostic Data Selection Bias](https://mlsanity.com/paper/1708.06656/) (2017)
- [Incorporating Unlabeled Data into Distributionally Robust Learning](https://mlsanity.com/paper/1912.07729/) (2019)

## Built on by (most cited first, 30 of 90)

- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Handling Distribution Shifts on Graphs: An Invariance Perspective](https://mlsanity.com/paper/2202.02466/) (2022)
- [Causality Inspired Representation Learning for Domain Generalization](https://mlsanity.com/paper/2203.14237/) (2022)
- [Causal Attention for Unbiased Visual Recognition](https://mlsanity.com/paper/2108.08782/) (2021)
- [ZIN: When and How to Learn Invariance Without Environment Partition?](https://mlsanity.com/paper/2203.05818/) (2022)
- [Out-of-distribution Generalization with Causal Invariant Transformations](https://mlsanity.com/paper/2203.11528/) (2022)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)
- [Does Invariant Graph Learning via Environment Augmentation Learn Invariance?](https://mlsanity.com/paper/2310.19035/) (2023)
- [Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift](https://mlsanity.com/paper/2211.02843/) (2022)
- [Identifying Weight-Variant Latent Causal Models](https://mlsanity.com/paper/2208.14153/) (2022)
- [Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization](https://mlsanity.com/paper/2306.01103/) (2023)
- [Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learning](https://mlsanity.com/paper/2406.09130/) (2024)
- [Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains](https://mlsanity.com/paper/2304.02950/) (2023)
- [Invariant Learning via Probability of Sufficient and Necessary Causes](https://mlsanity.com/paper/2309.12559/) (2023)
- [Kernelized Heterogeneous Risk Minimization](https://mlsanity.com/paper/2110.12425/) (2021)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [Rethinking the Evaluation Protocol of Domain Generalization](https://mlsanity.com/paper/2305.15253/) (2023)
- [A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization](https://mlsanity.com/paper/2111.02355/) (2021)
- [When Does Group Invariant Learning Survive Spurious Correlations?](https://mlsanity.com/paper/2206.14534/) (2022)
- [Exploring Optimal Substructure for Out-of-distribution Generalization via Feature-targeted Model Pruning](https://mlsanity.com/paper/2212.09458/) (2022)
- [Learning to Split for Automatic Bias Detection](https://mlsanity.com/paper/2204.13749/) (2022)
- [Universal Domain Adaptation via Compressive Attention Matching](https://mlsanity.com/paper/2304.11862/) (2023)
- [Domain Generalisation via Imprecise Learning](https://mlsanity.com/paper/2404.04669/) (2024)
- [Stable Learning via Sparse Variable Independence](https://mlsanity.com/paper/2212.00992/) (2022)
- [Multimodality Invariant Learning for Multimedia-Based New Item Recommendation](https://mlsanity.com/paper/2405.15783/) (2024)
- [An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation](https://mlsanity.com/paper/2205.12753/) (2022)
- [Towards out of distribution generalization for problems in mechanics](https://mlsanity.com/paper/2206.14917/) (2022)
- [Towards Robust Out-of-Distribution Generalization Bounds via Sharpness](https://mlsanity.com/paper/2403.06392/) (2024)
- [Graph Structure and Feature Extrapolation for Out-of-Distribution Generalization](https://mlsanity.com/paper/2306.08076/) (2023)

## Abstract (arXiv)

Machine learning algorithms with empirical risk minimization usually suffer from poor generalization performance due to the greedy exploitation of correlations among the training data, which are not stable under distributional shifts. Recently, some invariant learning methods for out-of-distribution (OOD) generalization have been proposed by leveraging multiple training environments to find invariant relationships. However, modern datasets are frequently assembled by merging data from multiple sources without explicit source labels. The resultant unobserved heterogeneity renders many invariant learning methods inapplicable. In this paper, we propose Heterogeneous Risk Minimization (HRM) framework to achieve joint learning of latent heterogeneity among the data and invariant relationship, which leads to stable prediction despite distributional shifts. We theoretically characterize the roles of the environment labels in invariant learning and justify our newly proposed HRM framework. Extensive experimental results validate the effectiveness of our HRM framework.