# Evaluating Treatment Benefit Predictors using Observational Data: Contending with Identification and Confounding Bias

arXiv: https://arxiv.org/abs/2407.05585 · HTML: https://mlsanity.com/paper/2407.05585/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2024.

## Built on by (most cited first, 1 of 1)

- [Recent Advances, Applications and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2024 Symposium](https://mlsanity.com/paper/2502.06693/) (2025)

## Abstract (arXiv)

A treatment benefit predictor (TBP) is a function that maps patient characteristics to an estimate of the treatment benefit for that patient. Such predictors support optimizing individualized treatment decisions, which are central to precision medicine. However, evaluating the predictive performance of a TBP is challenging, as this often must be conducted in a sample where treatment assignment is not random. After briefly reviewing several metrics for evaluating TBPs, we show conceptually how to evaluate a pre-specified TBP using observational data from the target population, for a binary treatment decision at a single time point. We exemplify with a particular measure of discrimination (the concentration of benefit index) and a particular measure of calibration (the moderate calibration curve). The population-level definitions of these metrics involve the latent treatment benefit variable, but we show identification by re-expressing the respective estimands in terms of the distribution of observable data only. We also show that in the absence of full confounding control, bias propagates in a more complex manner than when targeting more commonly encountered estimands. We find the patterns of biases are often unpredictable, and general intuition about the direction of bias in causal effect estimates does not hold in the present context.