# Diagnosing failures of fairness transfer across distribution shift in real-world medical settings

arXiv: https://arxiv.org/abs/2202.01034 · HTML: https://mlsanity.com/paper/2202.01034/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 101 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 30 of 49)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [To be Robust or to be Fair: Towards Fairness in Adversarial Training](https://mlsanity.com/paper/2010.06121/) (2020)
- [Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions](https://mlsanity.com/paper/1707.06422/) (2017)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests](https://mlsanity.com/paper/2106.00545/) (2021)
- [Training individually fair ML models with Sensitive Subspace Robustness](https://mlsanity.com/paper/1907.00020/) (2019)
- [Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions](https://mlsanity.com/paper/2104.03829/) (2021)
- [Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning](https://mlsanity.com/paper/2006.12621/) (2020)

## Built on by (most cited first, 30 of 45)

- [Generative models improve fairness of medical classifiers under distribution shifts](https://mlsanity.com/paper/2304.09218/) (2023)
- [MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models](https://mlsanity.com/paper/2410.13085/) (2024)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Supervised Algorithmic Fairness in Distribution Shifts: A Survey](https://mlsanity.com/paper/2402.01327/) (2024)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [No Fair Lunch: A Causal Perspective on Dataset Bias in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2307.16526/) (2023)
- [Designing Long-term Group Fair Policies in Dynamical Systems](https://mlsanity.com/paper/2311.12447/) (2023)
- [Mind the Graph When Balancing Data for Fairness or Robustness](https://mlsanity.com/paper/2406.17433/) (2024)
- [Rethinking Fair Representation Learning for Performance-Sensitive Tasks](https://mlsanity.com/paper/2410.04120/) (2024)
- [A unified framework for dataset shift diagnostics](https://mlsanity.com/paper/2205.08340/) (2022)
- [Algorithmic Fairness Generalization under Covariate and Dependence Shifts Simultaneously](https://mlsanity.com/paper/2311.13816/) (2023)
- [Explanation Shift: How Did the Distribution Shift Impact the Model?](https://mlsanity.com/paper/2303.08081/) (2023)
- [Proxy Methods for Domain Adaptation](https://mlsanity.com/paper/2403.07442/) (2024)
- [Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness](https://mlsanity.com/paper/2506.04193/) (2025)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks](https://mlsanity.com/paper/2307.07002/) (2023)
- [The Case for Globalizing Fairness: A Mixed Methods Study on Colonialism, AI, and Health in Africa](https://mlsanity.com/paper/2403.03357/) (2024)
- [Scalable Out-of-distribution Robustness in the Presence of Unobserved Confounders](https://mlsanity.com/paper/2411.19923/) (2024)
- [When Shift Happens - Confounding Is to Blame](https://mlsanity.com/paper/2505.21422/) (2025)
- [The Limits of Fair Medical Imaging AI In The Wild](https://mlsanity.com/paper/2312.10083/) (2023)
- [Looking Beyond What You See: An Empirical Analysis on Subgroup Intersectional Fairness for Multi-label Chest X-ray Classification Using Social Determinants of Racial Health Inequities](https://mlsanity.com/paper/2403.18196/) (2024)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)
- [Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neuroscience](https://mlsanity.com/paper/2308.03175/) (2023)
- [Regulating AI Adaptation: An Analysis of AI Medical Device Updates](https://mlsanity.com/paper/2407.16900/) (2024)
- [Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications](https://mlsanity.com/paper/2407.16953/) (2024)
- [Unsupervised bias discovery in medical image segmentation](https://mlsanity.com/paper/2309.00451/) (2023)
- [Mixture Data for Training Cannot Ensure Out-of-distribution Generalization](https://mlsanity.com/paper/2312.16243/) (2023)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [Using Backbone Foundation Model for Evaluating Fairness in Chest Radiography Without Demographic Data](https://mlsanity.com/paper/2408.16130/) (2024)
- [Understanding-informed Bias Mitigation for Fair CMR Segmentation](https://mlsanity.com/paper/2503.17089/) (2025)

## Abstract (arXiv)

Diagnosing and mitigating changes in model fairness under distribution shift is an important component of the safe deployment of machine learning in healthcare settings. Importantly, the success of any mitigation strategy strongly depends on the structure of the shift. Despite this, there has been little discussion of how to empirically assess the structure of a distribution shift that one is encountering in practice. In this work, we adopt a causal framing to motivate conditional independence tests as a key tool for characterizing distribution shifts. Using our approach in two medical applications, we show that this knowledge can help diagnose failures of fairness transfer, including cases where real-world shifts are more complex than is often assumed in the literature. Based on these results, we discuss potential remedies at each step of the machine learning pipeline.