# Stability of clinical prediction models developed using statistical or machine learning methods

arXiv: https://arxiv.org/abs/2211.01061 · HTML: https://mlsanity.com/paper/2211.01061/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 67 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 1 of 1)

- [To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets](https://mlsanity.com/paper/2101.11230/) (2021)

## Built on by (most cited first, 12 of 12)

- [Evaluating Model Explanations without Ground Truth](https://mlsanity.com/paper/2505.10399/) (2025)
- [Synthetic Data Generation for Augmenting Small Samples](https://mlsanity.com/paper/2501.18741/) (2025)
- [Critical Appraisal of Fairness Metrics in Clinical Predictive AI](https://mlsanity.com/paper/2506.17035/) (2025)
- [Understanding overfitting in random forest for probability estimation: a visualization and simulation study](https://mlsanity.com/paper/2402.18612/) (2024)
- [ROOFS: RObust biOmarker Feature Selection](https://mlsanity.com/paper/2601.05151/) (2026)
- [Reducing cross-sample prediction churn in scientific machine learning](https://mlsanity.com/paper/2605.13826/) (2026)
- [Speculations on Uncertainty and Humane Algorithms](https://mlsanity.com/paper/2408.06736/) (2024)
- [Masked Omics Modeling for Multimodal Representation Learning across Histopathology and Molecular Profiles](https://mlsanity.com/paper/2508.00969/) (2025)
- [Machine learning for violence prediction: a systematic review and critical appraisal](https://mlsanity.com/paper/2511.23118/) (2025)
- [Bootstrapping-based Regularisation for Reducing Individual Prediction Instability in Clinical Risk Prediction Models](https://mlsanity.com/paper/2602.11360/) (2026)
- [Diagnostics for Individual-Level Prediction Instability in Machine Learning for Healthcare](https://mlsanity.com/paper/2603.00192/) (2026)
- [A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models](https://mlsanity.com/paper/2606.00027/) (2026)

## Abstract (arXiv)

Clinical prediction models estimate an individual's risk of a particular health outcome, conditional on their values of multiple predictors. A developed model is a consequence of the development dataset and the chosen model building strategy, including the sample size, number of predictors and analysis method (e.g., regression or machine learning). Here, we raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Then, through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers should always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model building steps (those used in the development of the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and then deriving (i) a prediction instability plot of bootstrap model predictions (y-axis) versus original model predictions (x-axis), (ii) a calibration instability plot showing calibration curves for the bootstrap models in the original sample; and (iii) the instability index, which is the mean absolute difference between individuals' original and bootstrap model predictions. A case study is used to illustrate how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), whilst also informing a model's critical appraisal (risk of bias rating), fairness assessment and further validation requirements.