# Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations

arXiv: https://arxiv.org/abs/2211.15646 · HTML: https://mlsanity.com/paper/2211.15646/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 30 of 33)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Contrastive Test-Time Adaptation](https://mlsanity.com/paper/2204.10377/) (2022)
- [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021)
- [A Fine-Grained Analysis on Distribution Shift](https://mlsanity.com/paper/2110.11328/) (2021)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [On Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2210.11369/) (2022)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation](https://mlsanity.com/paper/2204.02070/) (2022)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations](https://mlsanity.com/paper/2107.00520/) (2021)
- [Invariant and Transportable Representations for Anti-Causal Domain Shifts](https://mlsanity.com/paper/2207.01603/) (2022)
- [Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization](https://mlsanity.com/paper/2206.07837/) (2022)
- [Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcuts](https://mlsanity.com/paper/2009.10132/) (2020)
- [BARACK: Partially Supervised Group Robustness With Guarantees](https://mlsanity.com/paper/2201.00072/) (2021)
- [Revisiting Self-Distillation](https://mlsanity.com/paper/2206.08491/) (2022)
- [Fairness and robustness in anti-causal prediction](https://mlsanity.com/paper/2209.09423/) (2022)

## Built on by (most cited first, 7 of 7)

- [The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning](https://mlsanity.com/paper/2402.12715/) (2024)
- [Substance or Style: What Does Your Image Embedding Know?](https://mlsanity.com/paper/2307.05610/) (2023)
- [Scalable Out-of-distribution Robustness in the Presence of Unobserved Confounders](https://mlsanity.com/paper/2411.19923/) (2024)
- [Modularity Trumps Invariance for Compositional Robustness](https://mlsanity.com/paper/2306.09005/) (2023)
- [Label Shift Estimators for Non-Ignorable Missing Data](https://mlsanity.com/paper/2310.18261/) (2023)
- [Autoencoder based approach for the mitigation of spurious correlations](https://mlsanity.com/paper/2406.18901/) (2024)
- [Theory-inspired Label Shift Adaptation via Aligned Distribution Mixture](https://mlsanity.com/paper/2411.02047/) (2024)

## Abstract (arXiv)

Changes in the data distribution at test time can have deleterious effects on the performance of predictive models p(y|x). We consider situations where there are additional meta-data labels (such as group labels), denoted by z, that can account for such changes in the distribution. In particular, we assume that the prior distribution p(y, z), which models the dependence between the class label y and the "nuisance" factors z, may change across domains, either due to a change in the correlation between these terms, or a change in one of their marginals. However, we assume that the generative model for features p(x|y,z) is invariant across domains. We note that this corresponds to an expanded version of the widely used "label shift" assumption, where the labels now also include the nuisance factors z. Based on this observation, we propose a test-time label shift correction that adapts to changes in the joint distribution p(y, z) using EM applied to unlabeled samples from the target domain distribution, pt(x). Importantly, we are able to avoid fitting a generative model p(x|y, z), and merely need to reweight the outputs of a discriminative model ps(y, z|x) trained on the source distribution. We evaluate our method, which we call "Test-Time Label-Shift Adaptation" (TTLSA), on several standard image and text datasets, as well as the CheXpert chest X-ray dataset, and show that it improves performance over methods that target invariance to changes in the distribution, as well as baseline empirical risk minimization methods. Code for reproducing experiments is available at https://github.com/nalzok/test-time-label-shift .