# In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation

arXiv: https://arxiv.org/abs/2302.02923 · HTML: https://mlsanity.com/paper/2302.02923/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms](https://mlsanity.com/paper/2101.10943/) (2021)
- [Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition](https://mlsanity.com/paper/1707.02641/) (2017)
- [Discussion on "Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects" by Hahn, Murray and Carvalho](https://mlsanity.com/paper/2108.02836/) (2021)
- [On Inductive Biases for Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2106.03765/) (2021)
- [Counterfactual Representation Learning with Balancing Weights](https://mlsanity.com/paper/2010.12618/) (2020)
- [Learning Weighted Representations for Generalization Across Designs](https://mlsanity.com/paper/1802.08598/) (2018)
- [Using Wasserstein Generative Adversarial Networks for the Design of Monte Carlo Simulations](https://mlsanity.com/paper/1909.02210/) (2019)
- [A comparison of methods for model selection when estimating individual treatment effects](https://mlsanity.com/paper/1804.05146/) (2018)
- [Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.01939/) (2022)
- [Validating Causal Inference Methods](https://mlsanity.com/paper/2202.04208/) (2022)
- [Synth-Validation: Selecting the Best Causal Inference Method for a Given Dataset](https://mlsanity.com/paper/1711.00083/) (2017)
- [Selective machine learning of doubly robust functionals](https://mlsanity.com/paper/1911.02029/) (2019)
- [β-Intact-VAE: Identifying and Estimating Causal Effects under Limited Overlap](https://mlsanity.com/paper/2110.05225/) (2021)

## Built on by (most cited first, 29 of 29)

- [Causal machine learning for predicting treatment outcomes](https://mlsanity.com/paper/2410.08770/) (2024)
- [DiffPO: A causal diffusion model for learning distributions of potential outcomes](https://mlsanity.com/paper/2410.08924/) (2024)
- [Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimation](https://mlsanity.com/paper/2305.15984/) (2023)
- [Quantifying Aleatoric Uncertainty of the Treatment Effect: A Novel Orthogonal Learner](https://mlsanity.com/paper/2411.03387/) (2024)
- [The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation](https://mlsanity.com/paper/2303.01412/) (2023)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Orthogonal Representation Learning for Estimating Causal Quantities](https://mlsanity.com/paper/2502.04274/) (2025)
- [Causal Q-Aggregation for CATE Model Selection](https://mlsanity.com/paper/2310.16945/) (2023)
- [Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choice](https://mlsanity.com/paper/2310.18212/) (2023)
- [Optimizing Treatment Allocation in the Presence of Interference](https://mlsanity.com/paper/2410.00075/) (2024)
- [Improving the Generation and Evaluation of Synthetic Data for Downstream Medical Causal Inference](https://mlsanity.com/paper/2510.18768/) (2025)
- [Transfer Learning for Causal Effect Estimation](https://mlsanity.com/paper/2305.09126/) (2023)
- [Learned Causal Method Prediction](https://mlsanity.com/paper/2311.03989/) (2023)
- [PairNet: Training with Observed Pairs to Estimate Individual Treatment Effect](https://mlsanity.com/paper/2406.03864/) (2024)
- [Federated Learning for Estimating Heterogeneous Treatment Effects](https://mlsanity.com/paper/2402.17705/) (2024)
- [Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators](https://mlsanity.com/paper/2402.18392/) (2024)
- [Defining Expertise: Applications to Treatment Effect Estimation](https://mlsanity.com/paper/2403.00694/) (2024)
- [Sources of Gain: Decomposing Performance in Conditional Average Dose Response Estimation](https://mlsanity.com/paper/2406.08206/) (2024)
- [Assessing Electricity Service Unfairness with Transfer Counterfactual Learning](https://mlsanity.com/paper/2310.03258/) (2023)
- [Overlap-Adaptive Regularization for Conditional Average Treatment Effect Estimation](https://mlsanity.com/paper/2509.24962/) (2025)
- [A Relative Error-Based Evaluation Framework of Heterogeneous Treatment Effect Estimators](https://mlsanity.com/paper/2510.16419/) (2025)
- [Estimating Treatment Effects in Networks under Unknown Exposure Mappings](https://mlsanity.com/paper/2510.21457/) (2025)
- [Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness](https://mlsanity.com/paper/2603.20775/) (2026)
- [Using representation balancing to learn conditional-average dose responses from clustered data](https://mlsanity.com/paper/2309.03731/) (2023)
- [Robust CATE Estimation Using Novel Ensemble Methods](https://mlsanity.com/paper/2407.03690/) (2024)
- [Heterogeneous Treatment Effect in Time-to-Event Outcomes: Harnessing Censored Data with Recursively Imputed Trees](https://mlsanity.com/paper/2502.01575/) (2025)
- [Leveraging a Simulator for Learning Causal Representations from Post-Treatment Covariates for CATE](https://mlsanity.com/paper/2502.05037/) (2025)
- [Bayesian X-Learner: Calibrated Posterior Inference for Heterogeneous Treatment Effects under Heavy-Tailed Outcomes](https://mlsanity.com/paper/2604.27394/) (2026)
- [Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation](https://mlsanity.com/paper/2605.10430/) (2026)

## Abstract (arXiv)

Personalized treatment effect estimates are often of interest in high-stakes applications -- thus, before deploying a model estimating such effects in practice, one needs to be sure that the best candidate from the ever-growing machine learning toolbox for this task was chosen. Unfortunately, due to the absence of counterfactual information in practice, it is usually not possible to rely on standard validation metrics for doing so, leading to a well-known model selection dilemma in the treatment effect estimation literature. While some solutions have recently been investigated, systematic understanding of the strengths and weaknesses of different model selection criteria is still lacking. In this paper, instead of attempting to declare a global `winner', we therefore empirically investigate success- and failure modes of different selection criteria. We highlight that there is a complex interplay between selection strategies, candidate estimators and the data used for comparing them, and provide interesting insights into the relative (dis)advantages of different criteria alongside desiderata for the design of further illuminating empirical studies in this context.