# When accurate prediction models yield harmful self-fulfilling prophecies

arXiv: https://arxiv.org/abs/2312.01210 · HTML: https://mlsanity.com/paper/2312.01210/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 48 references that are in the map.
- First posted on arXiv 2 Dec 2023.

## Builds on (most cited first, 8 of 8)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations](https://mlsanity.com/paper/2107.00520/) (2021)
- [Model updating after interventions paradoxically introduces bias](https://mlsanity.com/paper/2010.11530/) (2020)
- [Prediction under interventions: evaluation of counterfactual performance using longitudinal observational data](https://mlsanity.com/paper/2304.10005/) (2023)
- [From algorithms to action: improving patient care requires causality](https://mlsanity.com/paper/2209.07397/) (2022)

## Built on by (most cited first, 12 of 12)

- [Human Expertise in Algorithmic Prediction](https://mlsanity.com/paper/2402.00793/) (2024)
- [On Continuous Monitoring of Risk Violations under Unknown Shift](https://mlsanity.com/paper/2506.16416/) (2025)
- [Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets](https://mlsanity.com/paper/2406.06671/) (2024)
- [Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness](https://mlsanity.com/paper/2506.04193/) (2025)
- [Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts](https://mlsanity.com/paper/2507.19477/) (2025)
- [Integrating Expert Judgment and Algorithmic Decision Making: An Indistinguishability Framework](https://mlsanity.com/paper/2410.08783/) (2024)
- [Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions](https://mlsanity.com/paper/2510.21335/) (2025)
- [Inducing Causal Structure for Interpretable Neural Networks Applied to Glucose Prediction for T1DM Patients](https://mlsanity.com/paper/2503.14442/) (2025)
- [From Data-Driven to Purpose-Driven Artificial Intelligence: Systems Thinking for Data-Analytic Automation of Patient Care](https://mlsanity.com/paper/2506.13584/) (2025)
- [When Predictions Shape Reality: A Socio-Technical Synthesis of Performative Predictions in Machine Learning](https://mlsanity.com/paper/2601.04447/) (2026)
- [Actions Have Consequences: Detecting Outcome Performativity using Intervention Testing](https://mlsanity.com/paper/2607.26908/) (2026)
- [Large Language Models are Approximate Survival Estimators](https://mlsanity.com/paper/2609.38181/) (2026)

## Abstract (arXiv)

Prediction models are popular in medical research and practice. By predicting an outcome of interest for specific patients, these models may help inform difficult treatment decisions, and are often hailed as the poster children for personalized, data-driven healthcare. We show however, that using prediction models for decision making can lead to harmful decisions, even when the predictions exhibit good discrimination after deployment. These models are harmful self-fulfilling prophecies: their deployment harms a group of patients but the worse outcome of these patients does not invalidate the predictive power of the model. Our main result is a formal characterization of a set of such prediction models. Next we show that models that are well calibrated before and after deployment are useless for decision making as they made no change in the data distribution. These results point to the need to revise standard practices for validation, deployment and evaluation of prediction models that are used in medical decisions.