# Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging

arXiv: https://arxiv.org/abs/2403.04484 · HTML: https://mlsanity.com/paper/2403.04484/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Physics-based Noise Modeling for Extreme Low-light Photography](https://mlsanity.com/paper/2108.02158/) (2021)
- [What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors](https://mlsanity.com/paper/2203.01825/) (2022)
- [A Data-Based Perspective on Transfer Learning](https://mlsanity.com/paper/2207.05739/) (2022)
- [On the Connection between Pre-training Data Diversity and Fine-tuning Robustness](https://mlsanity.com/paper/2307.12532/) (2023)
- [UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classification](https://mlsanity.com/paper/2312.11038/) (2023)
- [The Role of Pre-training Data in Transfer Learning](https://mlsanity.com/paper/2302.13602/) (2023)
- [Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations?](https://mlsanity.com/paper/2307.12344/) (2023)
- [Cats or CAT scans: transfer learning from natural or medical image source datasets?](https://mlsanity.com/paper/1810.05444/) (2018)
- [Deep Learning for Distinguishing Normal versus Abnormal Chest Radiographs and Generalization to Unseen Diseases](https://mlsanity.com/paper/2010.11375/) (2020)
- [Robustness Stress Testing in Medical Image Classification](https://mlsanity.com/paper/2308.06889/) (2023)
- [Does Medical Imaging learn different Convolution Filters?](https://mlsanity.com/paper/2210.13799/) (2022)
- [Revisiting Hidden Representations in Transfer Learning for Medical Imaging](https://mlsanity.com/paper/2302.08272/) (2023)

## Built on by (most cited first, 10 of 10)

- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [General Methods Make Great Domain-specific Foundation Models: A Case-study on Fetal Ultrasound](https://mlsanity.com/paper/2506.19552/) (2025)
- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)
- [In-hoc Concept Representations to Regularise Deep Learning in Medical Imaging](https://mlsanity.com/paper/2508.13880/) (2025)
- [Exploring connections of spectral analysis and transfer learning in medical imaging](https://mlsanity.com/paper/2407.11379/) (2024)
- [SeagrassFinder: Deep Learning for Eelgrass Detection and Coverage Estimation in the Wild](https://mlsanity.com/paper/2412.16147/) (2024)
- [Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context](https://mlsanity.com/paper/2509.19671/) (2025)
- [Learning to Harmonize Cross-vendor X-ray Images by Non-linear Image Dynamics Correction](https://mlsanity.com/paper/2504.10080/) (2025)
- [Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis](https://mlsanity.com/paper/2505.16773/) (2025)
- [Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP](https://mlsanity.com/paper/2608.12086/) (2026)

## Abstract (arXiv)

Transfer learning has become an essential part of medical imaging classification algorithms, often leveraging ImageNet weights. The domain shift from natural to medical images has prompted alternatives such as RadImageNet, often showing comparable classification performance. However, it remains unclear whether the performance gains from transfer learning stem from improved generalization or shortcut learning. To address this, we conceptualize confounders by introducing the Medical Imaging Contextualized Confounder Taxonomy (MICCAT) and investigate a range of confounders across it -- whether synthetic or sampled from the data -- using two public chest X-ray and CT datasets. We show that ImageNet and RadImageNet achieve comparable classification performance, yet ImageNet is much more prone to overfitting to confounders. We recommend that researchers using ImageNet-pretrained models reexamine their model robustness by conducting similar experiments. Our code and experiments are available at https://github.com/DovileDo/source-matters.