# Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging

arXiv: https://arxiv.org/abs/1912.03606 · HTML: https://mlsanity.com/paper/1912.03606/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2019.

## Built on by (most cited first, 3 of 3)

- [Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex](https://mlsanity.com/paper/2211.14673/) (2022)
- [Model Selection's Disparate Impact in Real-World Deep Learning Applications](https://mlsanity.com/paper/2104.00606/) (2021)
- [Deep Learning for Automatic Detection and Facial Recognition in Japanese Macaques: Illuminating Social Networks](https://mlsanity.com/paper/2310.06489/) (2023)

## Abstract (arXiv)

We reproduced the results of CheXNet with fixed hyperparameters and 50 different random seeds to identify 14 finding in chest radiographs (x-rays). Because CheXNet fine-tunes a pre-trained DenseNet, the random seed affects the ordering of the batches of training data but not the initialized model weights. We found substantial variability in predictions for the same radiograph across model runs (mean ln[(maximum probability)/(minimum probability)] 2.45, coefficient of variation 0.543). This individual radiograph-level variability was not fully reflected in the variability of AUC on a large test set. Averaging predictions from 10 models reduced variability by nearly 70% (mean coefficient of variation from 0.543 to 0.169, t-test 15.96, p-value < 0.0001). We encourage researchers to be aware of the potential variability of CNNs and ensemble predictions from multiple models to minimize the effect this variability may have on the care of individual patients when these models are deployed clinically.