# How to select predictive models for causal inference?

arXiv: https://arxiv.org/abs/2302.00370 · HTML: https://mlsanity.com/paper/2302.00370/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 95 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition](https://mlsanity.com/paper/1707.02641/) (2017)
- [Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects](https://mlsanity.com/paper/2001.07426/) (2020)
- [Identifying Causal-Effect Inference Failure with Uncertainty-Aware Models](https://mlsanity.com/paper/2007.00163/) (2020)
- [Some methods for heterogeneous treatment effect estimation in high-dimensions](https://mlsanity.com/paper/1707.00102/) (2017)
- [Counterfactual Risk Minimization: Learning from Logged Bandit Feedback](https://mlsanity.com/paper/1502.02362/) (2015)
- [Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis](https://mlsanity.com/paper/1802.05046/) (2018)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [A comparison of methods for model selection when estimating individual treatment effects](https://mlsanity.com/paper/1804.05146/) (2018)
- [An Evaluation Toolkit to Guide Model Selection and Cohort Definition in Causal Inference](https://mlsanity.com/paper/1906.00442/) (2019)
- [Challenges in Obtaining Valid Causal Effect Estimates with Machine Learning Algorithms](https://mlsanity.com/paper/1711.07137/) (2017)
- [A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint](https://mlsanity.com/paper/2106.05588/) (2021)
- [Random Forests of Interaction Trees for Estimating Individualized Treatment Effects in Randomized Trials](https://mlsanity.com/paper/1709.04862/) (2017)
- [Assessment of Heterogeneous Treatment Effect Estimation Accuracy via Matching](https://mlsanity.com/paper/2003.03881/) (2020)

## Built on by (most cited first, 2 of 2)

- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [Learned Causal Method Prediction](https://mlsanity.com/paper/2311.03989/) (2023)

## Abstract (arXiv)

As predictive models -- e.g., from machine learning -- give likely outcomes, they may be used to reason on the effect of an intervention, a causal-inference task. The increasing complexity of health data has opened the door to a plethora of models, but also the Pandora box of model selection: which of these models yield the most valid causal estimates? Here we highlight that classic machine-learning model selection does not select the best outcome models for causal inference. Indeed, causal model selection should control both outcome errors for each individual, treated or not treated, whereas only one outcome is observed. Theoretically, simple risks used in machine learning do not control causal effects when treated and non-treated population differ too much. More elaborate risks build proxies of the causal error using ``nuisance'' re-weighting to compute it on the observed data. But does computing these nuisance adds noise to model selection? Drawing from an extensive empirical study, we outline a good causal model-selection procedure: using the so-called R-risk; using flexible estimators to compute the nuisance models on the train set; and splitting out 10% of the data to compute risks.