# Explaining medical AI performance disparities across sites with confounder Shapley value analysis

arXiv: https://arxiv.org/abs/2111.08168 · HTML: https://mlsanity.com/paper/2111.08168/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [On instabilities of deep learning in image reconstruction - Does AI come at a cost?](https://mlsanity.com/paper/1902.05300/) (2019)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [A Distributional Framework for Data Valuation](https://mlsanity.com/paper/2002.12334/) (2020)
- [Understanding Bias in Machine Learning](https://mlsanity.com/paper/1909.01866/) (2019)
- [An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classification](https://mlsanity.com/paper/2005.03912/) (2020)

## Built on by (most cited first, 7 of 7)

- ["Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts](https://mlsanity.com/paper/2210.10769/) (2022)
- [A hierarchical decomposition for explaining ML performance discrepancies](https://mlsanity.com/paper/2402.14254/) (2024)
- [Regulating AI Adaptation: An Analysis of AI Medical Device Updates](https://mlsanity.com/paper/2407.16900/) (2024)
- [Robustness of an Artificial Intelligence Solution for Diagnosis of Normal Chest X-Rays](https://mlsanity.com/paper/2209.09204/) (2022)
- [Real-World Performance of Autonomously Reporting Normal Chest Radiographs in NHS Trusts Using a Deep-Learning Algorithm on the GP Pathway](https://mlsanity.com/paper/2306.16115/) (2023)
- ["Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift](https://mlsanity.com/paper/2506.00756/) (2025)
- [Shapley Value Estimation for Multi-Site Data with Blockwise-Missing Features](https://mlsanity.com/paper/2609.14902/) (2026)

## Abstract (arXiv)

Medical AI algorithms can often experience degraded performance when evaluated on previously unseen sites. Addressing cross-site performance disparities is key to ensuring that AI is equitable and effective when deployed on diverse patient populations. Multi-site evaluations are key to diagnosing such disparities as they can test algorithms across a broader range of potential biases such as patient demographics, equipment types, and technical parameters. However, such tests do not explain why the model performs worse. Our framework provides a method for quantifying the marginal and cumulative effect of each type of bias on the overall performance difference when a model is evaluated on external data. We demonstrate its usefulness in a case study of a deep learning model trained to detect the presence of pneumothorax, where our framework can help explain up to 60% of the discrepancy in performance across different sites with known biases like disease comorbidities and imaging parameters.