# Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients

arXiv: https://arxiv.org/abs/2502.18975 · HTML: https://mlsanity.com/paper/2502.18975/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Learning to Diversify for Single Domain Generalization](https://mlsanity.com/paper/2108.11726/) (2021)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation](https://mlsanity.com/paper/2204.02070/) (2022)
- [Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2106.12723/) (2021)
- [On Guiding Visual Attention with Language Specification](https://mlsanity.com/paper/2202.08926/) (2022)
- [Counterfactual Image Editing](https://mlsanity.com/paper/2403.09683/) (2024)
- [Nuisances via Negativa: Adjusting for Spurious Correlations via Data Augmentation](https://mlsanity.com/paper/2210.01302/) (2022)
- [Studying How to Efficiently and Effectively Guide Models with Explanations](https://mlsanity.com/paper/2303.11932/) (2023)
- [Domain Generalization via Rationale Invariance](https://mlsanity.com/paper/2308.11158/) (2023)
- [Enabling Pedestrian Safety using Computer Vision Techniques: A Case Study of the 2018 Uber Inc. Self-driving Car Crash](https://mlsanity.com/paper/1805.11815/) (2018)
- [Automated wildlife image classification: An active learning tool for ecological applications](https://mlsanity.com/paper/2303.15823/) (2023)
- [Robustness through Data Augmentation Loss Consistency](https://mlsanity.com/paper/2110.11205/) (2021)
- [Meta-Learned Invariant Risk Minimization](https://mlsanity.com/paper/2103.12947/) (2021)
- [Shape-Biased Domain Generalization via Shock Graph Embeddings](https://mlsanity.com/paper/2109.05671/) (2021)
- [Constraint Guided Gradient Descent: Guided Training with Inequality Constraints](https://mlsanity.com/paper/2206.06202/) (2022)
- [Trust Me, I Know the Way: Predictive Uncertainty in the Presence of Shortcut Learning](https://mlsanity.com/paper/2502.09137/) (2025)

## Abstract (arXiv)

Machine learning models are inherently bound to the distribution of the training data, often exploiting non-causal shortcuts. As a result, achieving robustness to spurious correlations remains a challenge. While existing approaches rely on data manipulation or re-weighting strategies to achieve robustness, they typically require dense group labels, multiple training domains, or specialized pre-processing. We propose Invariance Pair Guidance (IPG), a method to mitigate reliance on spurious correlations using a sparse set of counterfactual pairs. Unlike other methods demanding extensive supervision, IPG utilizes a novel dual-update mechanism to dynamically correct the optimization trajectory. We generate input pairs that isolate the spurious attribute to define the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we define a corrective gradient that complements the traditional gradient descent approach. The correction adapts via a predefined invariance condition. Experiments on ColoredMNIST, Waterbirds-100, and CelebA datasets demonstrate the effectiveness of our approach and its robustness to group shifts, supported by a theoretical convergence analysis. IPG offers a data-efficient and theoretically grounded path to robustness.