# Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcuts

arXiv: https://arxiv.org/abs/2009.10132 · HTML: https://mlsanity.com/paper/2009.10132/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 29 of 29)

- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [Distilling Model Failures as Directions in Latent Space](https://mlsanity.com/paper/2206.14754/) (2022)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation](https://mlsanity.com/paper/2312.14223/) (2023)
- [FACTS: First Amplify Correlations and Then Slice to Discover Bias](https://mlsanity.com/paper/2309.17430/) (2023)
- [Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers](https://mlsanity.com/paper/2211.15231/) (2022)
- [The Role of Subgroup Separability in Group-Fair Medical Image Classification](https://mlsanity.com/paper/2307.02791/) (2023)
- [Fairness and robustness in anti-causal prediction](https://mlsanity.com/paper/2209.09423/) (2022)
- [Invariant Learning via Diffusion Dreamed Distribution Shifts](https://mlsanity.com/paper/2211.10370/) (2022)
- [Disparate Censorship & Undertesting: A Source of Label Bias in Clinical Machine Learning](https://mlsanity.com/paper/2208.01127/) (2022)
- [Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations](https://mlsanity.com/paper/2211.15646/) (2022)
- [Towards Dog Bark Decoding: Leveraging Human Speech Processing for Automated Bark Classification](https://mlsanity.com/paper/2404.18739/) (2024)
- [Patch Shortcuts: Interpretable Proxy Models Efficiently Find Black-Box Vulnerabilities](https://mlsanity.com/paper/2104.11691/) (2021)
- [Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models](https://mlsanity.com/paper/2304.03218/) (2023)
- [Looking Beyond What You See: An Empirical Analysis on Subgroup Intersectional Fairness for Multi-label Chest X-ray Classification Using Social Determinants of Racial Health Inequities](https://mlsanity.com/paper/2403.18196/) (2024)
- [Optimizing importance weighting in the presence of sub-population shifts](https://mlsanity.com/paper/2410.14315/) (2024)
- [Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models](https://mlsanity.com/paper/2506.00134/) (2025)
- [Learning disentangled representations for explainable chest X-ray classification using Dirichlet VAEs](https://mlsanity.com/paper/2302.02979/) (2023)
- [DEPICT: Diffusion-Enabled Permutation Importance for Image Classification Tasks](https://mlsanity.com/paper/2407.14509/) (2024)
- [X-Mark: Saliency-Guided Robust Dataset Ownership Verification for Medical Imaging](https://mlsanity.com/paper/2602.09284/) (2026)
- [Fair Machine Learning in Healthcare: A Review](https://mlsanity.com/paper/2206.14397/) (2022)
- [Detecting Dataset Bias in Medical AI: A Generalized and Modality-Agnostic Auditing Framework](https://mlsanity.com/paper/2503.09969/) (2025)
- [Weakly Supervised Tuberculosis Localization in Chest X-rays through Knowledge Distillation](https://mlsanity.com/paper/2512.11057/) (2025)
- [Weight Space Correlation Analysis: Quantifying Feature Utilization in Deep Learning Models](https://mlsanity.com/paper/2512.13144/) (2025)
- [Pitfalls of Administrative Censoring in Survival Models with Time-Indexed Inputs](https://mlsanity.com/paper/2607.10466/) (2026)
- [Cross-dataset transportability of pediatric chest X-ray deep learning across three countries: discrimination, calibration, operating-point failure, and limited-label recovery](https://mlsanity.com/paper/2609.05140/) (2026)

## Abstract (arXiv)

While deep learning has shown promise in improving the automated diagnosis of disease based on chest X-rays, deep networks may exhibit undesirable behavior related to shortcuts. This paper studies the case of spurious class skew in which patients with a particular attribute are spuriously more likely to have the outcome of interest. For instance, clinical protocols might lead to a dataset in which patients with pacemakers are disproportionately likely to have congestive heart failure. This skew can lead to models that take shortcuts by heavily relying on the biased attribute. We explore this problem across a number of attributes in the context of diagnosing the cause of acute hypoxemic respiratory failure. Applied to chest X-rays, we show that i) deep nets can accurately identify many patient attributes including sex (AUROC = 0.96) and age (AUROC >= 0.90), ii) they tend to exploit correlations between such attributes and the outcome label when learning to predict a diagnosis, leading to poor performance when such correlations do not hold in the test population (e.g., everyone in the test set is male), and iii) a simple transfer learning approach is surprisingly effective at preventing the shortcut and promoting good generalization performance. On the task of diagnosing congestive heart failure based on a set of chest X-rays skewed towards older patients (age >= 63), the proposed approach improves generalization over standard training from 0.66 (95% CI: 0.54-0.77) to 0.84 (95% CI: 0.73-0.92) AUROC. While simple, the proposed approach has the potential to improve the performance of models across populations by encouraging reliance on clinically relevant manifestations of disease, i.e., those that a clinician would use to make a diagnosis.