# Model updating after interventions paradoxically introduces bias

arXiv: https://arxiv.org/abs/2010.11530 · HTML: https://mlsanity.com/paper/2010.11530/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [When accurate prediction models yield harmful self-fulfilling prophecies](https://mlsanity.com/paper/2312.01210/) (2023)
- [Online covariance estimation for stochastic gradient descent under Markovian sampling](https://mlsanity.com/paper/2308.01481/) (2023)
- [Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees](https://mlsanity.com/paper/2110.06866/) (2021)
- [Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens](https://mlsanity.com/paper/2311.11463/) (2023)
- [Monitoring machine learning (ML)-based risk prediction algorithms in the presence of confounding medical interventions](https://mlsanity.com/paper/2211.09781/) (2022)
- [MLHOps: Machine Learning for Healthcare Operations](https://mlsanity.com/paper/2305.02474/) (2023)
- [Ethical considerations of use of hold-out sets in clinical prediction model management](https://mlsanity.com/paper/2406.03161/) (2024)
- [CLAM: Causal Spatial Disaggregation to Infer Local Effects From Coarse Data](https://mlsanity.com/paper/2608.08064/) (2026)
- [Actions Have Consequences: Detecting Outcome Performativity using Intervention Testing](https://mlsanity.com/paper/2607.26908/) (2026)

## Abstract (arXiv)

Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such `naive updating' when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues.