# Prediction under interventions: evaluation of counterfactual performance using longitudinal observational data

arXiv: https://arxiv.org/abs/2304.10005 · HTML: https://mlsanity.com/paper/2304.10005/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2023.

## Built on by (most cited first, 8 of 8)

- [Causal machine learning for predicting treatment outcomes](https://mlsanity.com/paper/2410.08770/) (2024)
- [When accurate prediction models yield harmful self-fulfilling prophecies](https://mlsanity.com/paper/2312.01210/) (2023)
- [Position: Human-Centric AI Requires a Minimum Viable Level of Human Understanding](https://mlsanity.com/paper/2602.00854/) (2026)
- [Forests for Differences: Robust Causal Inference Beyond Parametric DiD](https://mlsanity.com/paper/2505.09706/) (2025)
- [Measuring Model Performance in the Presence of an Intervention](https://mlsanity.com/paper/2511.05805/) (2025)
- [Debiased Machine Learning for Conformal Prediction of Counterfactual Outcomes Under Runtime Confounding](https://mlsanity.com/paper/2604.03772/) (2026)
- [An Auditable Policy-Simulation Framework for Student Dropout in Intervention-Free Data](https://mlsanity.com/paper/2604.08874/) (2026)
- [Physiological World Models for Human State Transitions](https://mlsanity.com/paper/2608.15309/) (2026)

## Abstract (arXiv)

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision making. However, evaluating predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different to those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting which involves creating a validation dataset that mimics the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.