# A comparison of methods for model selection when estimating individual treatment effects

arXiv: https://arxiv.org/abs/1804.05146 · HTML: https://mlsanity.com/paper/1804.05146/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2018.

## Built on by (most cited first, 28 of 28)

- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2302.02923/) (2023)
- [Causal Decision Making and Causal Effect Estimation Are Not the Same... and Why It Matters](https://mlsanity.com/paper/2104.04103/) (2021)
- [Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.01939/) (2022)
- [DoWhy: Addressing Challenges in Expressing and Validating Causal Assumptions](https://mlsanity.com/paper/2108.13518/) (2021)
- [Shrinkage Bayesian Causal Forests for Heterogeneous Treatment Effects Estimation](https://mlsanity.com/paper/2102.06573/) (2021)
- [Efficient Heterogeneous Treatment Effect Estimation With Multiple Experiments and Multiple Outcomes](https://mlsanity.com/paper/2206.04907/) (2022)
- [The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation](https://mlsanity.com/paper/2303.01412/) (2023)
- [Selecting Treatment Effects Models for Domain Adaptation Using Causal Knowledge](https://mlsanity.com/paper/2102.06271/) (2021)
- [Meta-Analysis of Randomized Experiments with Applications to Heavy-Tailed Response Data](https://mlsanity.com/paper/2112.07602/) (2021)
- [Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance](https://mlsanity.com/paper/2201.12692/) (2022)
- [Ensemble Method for Estimating Individualized Treatment Effects](https://mlsanity.com/paper/2202.12445/) (2022)
- [Causal Q-Aggregation for CATE Model Selection](https://mlsanity.com/paper/2310.16945/) (2023)
- [Learned Causal Method Prediction](https://mlsanity.com/paper/2311.03989/) (2023)
- [Improving Data-driven Heterogeneous Treatment Effect Estimation Under Structure Uncertainty](https://mlsanity.com/paper/2206.12689/) (2022)
- [How to select predictive models for causal inference?](https://mlsanity.com/paper/2302.00370/) (2023)
- [Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators](https://mlsanity.com/paper/2402.18392/) (2024)
- [Sources of Gain: Decomposing Performance in Conditional Average Dose Response Estimation](https://mlsanity.com/paper/2406.08206/) (2024)
- [Data-Driven Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/2301.06615/) (2023)
- [Causal Dynamic Variational Autoencoder for Counterfactual Regression in Longitudinal Data](https://mlsanity.com/paper/2310.10559/) (2023)
- [Cross-World Assumption and Refining Prediction Intervals for Individual Treatment Effects](https://mlsanity.com/paper/2507.12581/) (2025)
- [A pragmatic approach to estimating average treatment effects from EHR data: the effect of prone positioning on mechanically ventilated COVID-19 patients](https://mlsanity.com/paper/2109.06707/) (2021)
- [Out-of-sample scoring and automatic selection of causal estimators](https://mlsanity.com/paper/2212.10076/) (2022)
- [Orthogonal Series Estimation for the Ratio of Conditional Expectation Functions](https://mlsanity.com/paper/2212.13145/) (2022)
- [Robust CATE Estimation Using Novel Ensemble Methods](https://mlsanity.com/paper/2407.03690/) (2024)
- [Honesty in Causal Forests: When It Helps and When It Hurts](https://mlsanity.com/paper/2506.13107/) (2025)
- [Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation](https://mlsanity.com/paper/2605.10430/) (2026)

## Abstract (arXiv)

Practitioners in medicine, business, political science, and other fields are increasingly aware that decisions should be personalized to each patient, customer, or voter. A given treatment (e.g. a drug or advertisement) should be administered only to those who will respond most positively, and certainly not to those who will be harmed by it. Individual-level treatment effects can be estimated with tools adapted from machine learning, but different models can yield contradictory estimates. Unlike risk prediction models, however, treatment effect models cannot be easily evaluated against each other using a held-out test set because the true treatment effect itself is never directly observed. Besides outcome prediction accuracy, several metrics that can leverage held-out data to evaluate treatment effects models have been proposed, but they are not widely used. We provide a didactic framework that elucidates the relationships between the different approaches and compare them all using a variety of simulations of both randomized and observational data. Our results show that researchers estimating heterogenous treatment effects need not limit themselves to a single model-fitting algorithm. Instead of relying on a single method, multiple models fit by a diverse set of algorithms should be evaluated against each other using an objective function learned from the validation set. The model minimizing that objective should be used for estimating the individual treatment effect for future individuals.