# Improving the repeatability of deep learning models with Monte Carlo dropout

arXiv: https://arxiv.org/abs/2202.07562 · HTML: https://mlsanity.com/paper/2202.07562/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 71 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Rank consistent ordinal regression for neural networks with application to age estimation](https://mlsanity.com/paper/1901.07884/) (2019)
- [Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker](https://mlsanity.com/paper/1612.02572/) (2016)
- [Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs: A Deep Learning-Based Approach](https://mlsanity.com/paper/1710.10589/) (2017)
- [Breast Mass Classification from Mammograms using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1612.00542/) (2016)
- [Repeatability of Multiparametric Prostate MRI Radiomics Features](https://mlsanity.com/paper/1807.06089/) (2018)
- [FatSegNet : A Fully Automated Deep Learning Pipeline for Adipose Tissue Segmentation on Abdominal Dixon MRI](https://mlsanity.com/paper/1904.02082/) (2019)
- [SkiNet: A Deep Learning Solution for Skin Lesion Diagnosis with Uncertainty Estimation and Explainability](https://mlsanity.com/paper/2012.15049/) (2020)

## Built on by (most cited first, 11 of 11)

- [FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare](https://mlsanity.com/paper/2309.12325/) (2023)
- [Uncertainty-aware Language Modeling for Selective Question Answering](https://mlsanity.com/paper/2311.15451/) (2023)
- [Patient-specific, mechanistic models of tumor growth incorporating artificial intelligence and big data](https://mlsanity.com/paper/2308.14925/) (2023)
- [Controlled Dropout for Uncertainty Estimation](https://mlsanity.com/paper/2205.03109/) (2022)
- [Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis](https://mlsanity.com/paper/2505.03165/) (2025)
- [Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)](https://mlsanity.com/paper/2505.11738/) (2025)
- [Data-Driven Prediction and Uncertainty Quantification of PWR Crud-Induced Power Shift Using Convolutional Neural Networks](https://mlsanity.com/paper/2407.04726/) (2024)
- [Defining Replicability of Prediction Rules](https://mlsanity.com/paper/2305.01518/) (2023)
- [Training Variation of Physically-Informed Deep Learning Models](https://mlsanity.com/paper/2510.03416/) (2025)
- [Multi-Rater Calibrated Segmentation Models](https://mlsanity.com/paper/2605.02437/) (2026)
- [Adaptive Conformal Prediction for Image Regression Models with Application to an Inertial Confinement Fusion Emulator](https://mlsanity.com/paper/2610.00535/) (2026)

## Abstract (arXiv)

The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Repeatable models output predictions with low variation during independent tests carried out under similar conditions. During model development and evaluation, much attention is given to classification performance while model repeatability is rarely assessed, leading to the development of models that are unusable in clinical practice. In this work, we evaluate the repeatability of four model types (binary classification, multi-class classification, ordinal classification, and regression) on images that were acquired from the same patient during the same visit. We study the performance of binary, multi-class, ordinal, and regression models on four medical image classification tasks from public and private datasets: knee osteoarthritis, cervical cancer screening, breast density estimation, and retinopathy of prematurity. Repeatability is measured and compared on ResNet and DenseNet architectures. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increased repeatability for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95% limits of agreement by 16% points and of the disagreement rate by 7% points. The classification accuracy improved in most settings along with the repeatability. Our results suggest that beyond about 20 Monte Carlo iterations, there is no further gain in repeatability. In addition to the higher test-retest agreement, Monte Carlo predictions were better calibrated which leads to output probabilities reflecting more accurately the true likelihood of being correctly classified.