# Towards Understanding Variants of Invariant Risk Minimization through the Lens of Calibration

arXiv: https://arxiv.org/abs/2401.17541 · HTML: https://mlsanity.com/paper/2401.17541/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Domain Generalization for Object Recognition with Multi-task Autoencoders](https://mlsanity.com/paper/1508.07680/) (2015)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.03721/) (2021)
- [Out of Distribution Generalization in Machine Learning](https://mlsanity.com/paper/2103.02667/) (2021)
- [Optimal Representations for Covariate Shift](https://mlsanity.com/paper/2201.00057/) (2021)
- [Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalization](https://mlsanity.com/paper/2206.07766/) (2022)
- [Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning](https://mlsanity.com/paper/2003.12537/) (2020)
- [What Is Missing in IRM Training and Evaluation? Challenges and Solutions](https://mlsanity.com/paper/2303.02343/) (2023)
- [The Missing Invariance Principle Found -- the Reciprocal Twin of Invariant Risk Minimization](https://mlsanity.com/paper/2205.14546/) (2022)
- [An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration](https://mlsanity.com/paper/2307.08187/) (2023)

## Built on by (most cited first, 3 of 3)

- [TIF: Learning Temporal Invariance in Android Malware Detectors](https://mlsanity.com/paper/2502.05098/) (2025)
- [Robust Invariant Representation Learning by Distribution Extrapolation](https://mlsanity.com/paper/2505.16126/) (2025)
- [DeconDTN-Toolkit: A Library for Evaluation and Enhancement of Robustness to Provenance Shift](https://mlsanity.com/paper/2605.11237/) (2026)

## Abstract (arXiv)

Machine learning models traditionally assume that training and test data are independently and identically distributed. However, in real-world applications, the test distribution often differs from training. This problem, known as out-of-distribution (OOD) generalization, challenges conventional models. Invariant Risk Minimization (IRM) emerges as a solution that aims to identify invariant features across different environments to enhance OOD robustness. However, IRM's complexity, particularly its bi-level optimization, has led to the development of various approximate methods. Our study investigates these approximate IRM techniques, using the consistency and variance of calibration across environments as metrics to measure the invariance aimed for by IRM. Calibration, which measures the reliability of model prediction, serves as an indicator of whether models effectively capture environment-invariant features by showing how uniformly over-confident the model remains across varied environments. Through a comparative analysis of datasets with distributional shifts, we observe that Information Bottleneck-based IRM achieves consistent calibration across different environments. This observation suggests that information compression techniques, such as IB, are potentially effective in achieving model invariance. Furthermore, our empirical evidence indicates that models exhibiting consistent calibration across environments are also well-calibrated. This demonstrates that invariance and cross-environment calibration are empirically equivalent. Additionally, we underscore the necessity for a systematic approach to evaluating OOD generalization. This approach should move beyond traditional metrics, such as accuracy and F1 scores, which fail to account for the model's degree of over-confidence, and instead focus on the nuanced interplay between accuracy, calibration, and model invariance.