# Confounding variables can degrade generalization performance of radiological deep learning models

arXiv: https://arxiv.org/abs/1807.00431 · HTML: https://mlsanity.com/paper/1807.00431/

## Key facts

- Cited by 379 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2018.

## Built on by (most cited first, 30 of 379)

- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [On Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2210.11369/) (2022)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Causality-inspired Single-source Domain Generalization for Medical Image Segmentation](https://mlsanity.com/paper/2111.12525/) (2021)
- [Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization](https://mlsanity.com/paper/2212.10445/) (2022)
- [MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation](https://mlsanity.com/paper/2110.01406/) (2021)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Federated Learning Enables Big Data for Rare Cancer Boundary Detection](https://mlsanity.com/paper/2204.10836/) (2022)
- [IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography](https://mlsanity.com/paper/2103.12308/) (2021)
- [On the Impact of Spurious Correlation for Out-of-distribution Detection](https://mlsanity.com/paper/2109.05642/) (2021)
- [Simple and Fast Group Robustness by Automatic Feature Reweighting](https://mlsanity.com/paper/2306.11074/) (2023)
- [Probable Domain Generalization via Quantile Risk Minimization](https://mlsanity.com/paper/2207.09944/) (2022)
- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations](https://mlsanity.com/paper/2103.01890/) (2021)
- [Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2103.04244/) (2021)
- [OpenFL: An open-source framework for Federated Learning](https://mlsanity.com/paper/2105.06413/) (2021)
- [Towards Last-layer Retraining for Group Robustness with Fewer Annotations](https://mlsanity.com/paper/2309.08534/) (2023)
- [Do Input Gradients Highlight Discriminative Features?](https://mlsanity.com/paper/2102.12781/) (2021)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [The Federated Tumor Segmentation (FeTS) Challenge](https://mlsanity.com/paper/2105.05874/) (2021)
- [Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration](https://mlsanity.com/paper/2107.05446/) (2021)
- [Diagnosing failures of fairness transfer across distribution shift in real-world medical settings](https://mlsanity.com/paper/2202.01034/) (2022)
- [Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations](https://mlsanity.com/paper/2107.00520/) (2021)

## Abstract (arXiv)

Early results in using convolutional neural networks (CNNs) on x-rays to diagnose disease have been promising, but it has not yet been shown that models trained on x-rays from one hospital or one group of hospitals will work equally well at different hospitals. Before these tools are used for computer-aided diagnosis in real-world clinical settings, we must verify their ability to generalize across a variety of hospital systems. A cross-sectional design was used to train and evaluate pneumonia screening CNNs on 158,323 chest x-rays from NIH (n=112,120 from 30,805 patients), Mount Sinai (42,396 from 12,904 patients), and Indiana (n=3,807 from 3,683 patients). In 3 / 5 natural comparisons, performance on chest x-rays from outside hospitals was significantly lower than on held-out x-rays from the original hospital systems. CNNs were able to detect where an x-ray was acquired (hospital system, hospital department) with extremely high accuracy and calibrate predictions accordingly. The performance of CNNs in diagnosing diseases on x-rays may reflect not only their ability to identify disease-specific imaging findings on x-rays, but also their ability to exploit confounding information. Estimates of CNN performance based on test data from hospital systems used for model training may overstate their likely real-world performance.