# Ethical considerations of use of hold-out sets in clinical prediction model management

arXiv: https://arxiv.org/abs/2406.03161 · HTML: https://mlsanity.com/paper/2406.03161/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 35 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 3 of 3)

- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Algorithm Fairness in AI for Medicine and Healthcare](https://mlsanity.com/paper/2110.00603/) (2021)
- [Model updating after interventions paradoxically introduces bias](https://mlsanity.com/paper/2010.11530/) (2020)

## Built on by (most cited first, 1 of 1)

- [When Predictions Shape Reality: A Socio-Technical Synthesis of Performative Predictions in Machine Learning](https://mlsanity.com/paper/2601.04447/) (2026)

## Abstract (arXiv)

Clinical prediction models are statistical or machine learning models used to quantify the risk of a certain health outcome using patient data. These can then inform potential interventions on patients, causing an effect called performative prediction: predictions inform interventions which influence the outcome they were trying to predict, leading to a potential underestimation of risk in some patients if a model is updated on this data. One suggested resolution to this is the use of hold-out sets, in which a set of patients do not receive model derived risk scores, such that a model can be safely retrained. We present an overview of clinical and research ethics regarding potential implementation of hold-out sets for clinical prediction models in health settings. We focus on the ethical principles of beneficence, non-maleficence, autonomy and justice. We also discuss informed consent, clinical equipoise, and truth-telling. We present illustrative cases of potential hold-out set implementations and discuss statistical issues arising from different hold-out set sampling methods. We also discuss differences between hold-out sets and randomised control trials, in terms of ethics and statistical issues. Finally, we give practical recommendations for researchers interested in the use hold-out sets for clinical prediction models.