# Improving uplift model evaluation on RCT data

arXiv: https://arxiv.org/abs/2210.02152 · HTML: https://mlsanity.com/paper/2210.02152/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Uplift Modeling with Multiple Treatments and General Response Types](https://mlsanity.com/paper/1705.08492/) (2017)
- [Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects](https://mlsanity.com/paper/2111.07966/) (2021)
- [Learning to rank for uplift modeling](https://mlsanity.com/paper/2002.05897/) (2020)
- [Machine Learning for Variance Reduction in Online Experiments](https://mlsanity.com/paper/2106.07263/) (2021)
- [Response Transformation and Profit Decomposition for Revenue Uplift Modeling](https://mlsanity.com/paper/1911.08729/) (2019)
- [A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling](https://mlsanity.com/paper/2111.10106/) (2021)
- [High-dimensional regression adjustments in randomized experiments](https://mlsanity.com/paper/1607.06801/) (2016)
- [Towards Optimal Variance Reduction in Online Controlled Experiments](https://mlsanity.com/paper/2110.13406/) (2021)
- [A general framework for causal classification](https://mlsanity.com/paper/2003.11940/) (2020)
- [Affordable Uplift: Supervised Randomization in Controlled Experiments](https://mlsanity.com/paper/1910.00393/) (2019)
- [Estimating causal effects with optimization-based methods: A review and empirical comparison](https://mlsanity.com/paper/2203.00097/) (2022)

## Abstract (arXiv)

Estimating treatment effects is one of the most challenging and important tasks of data analysts. In many applications, like online marketing and personalized medicine, treatment needs to be allocated to the individuals where it yields a high positive treatment effect. Uplift models help select the right individuals for treatment and maximize the overall treatment effect (uplift). A major challenge in uplift modeling concerns model evaluation. Previous literature suggests methods like the Qini curve and the transformed outcome mean squared error. However, these metrics suffer from variance: their evaluations are strongly affected by random noise in the data, which renders their signals, to a certain degree, arbitrary. We theoretically analyze the variance of uplift evaluation metrics and derive possible methods of variance reduction, which are based on statistical adjustment of the outcome. We derive simple conditions under which the variance reduction methods improve the uplift evaluation metrics and empirically demonstrate their benefits on simulated and real-world data. Our paper provides strong evidence in favor of applying the suggested variance reduction procedures by default when evaluating uplift models on RCT data.