# The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation

arXiv: https://arxiv.org/abs/2303.01412 · HTML: https://mlsanity.com/paper/2303.01412/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [A Survey on Causal Inference](https://mlsanity.com/paper/2002.02770/) (2020)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [RealCause: Realistic Causal Inference Benchmarking](https://mlsanity.com/paper/2011.15007/) (2020)
- [Using Wasserstein Generative Adversarial Networks for the Design of Monte Carlo Simulations](https://mlsanity.com/paper/1909.02210/) (2019)
- [In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2302.02923/) (2023)
- [A comparison of methods for model selection when estimating individual treatment effects](https://mlsanity.com/paper/1804.05146/) (2018)
- [Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.01939/) (2022)
- [An Evaluation Toolkit to Guide Model Selection and Cohort Definition in Causal Inference](https://mlsanity.com/paper/1906.00442/) (2019)
- [Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure Learning Algorithms](https://mlsanity.com/paper/2107.03863/) (2021)
- [Synth-Validation: Selecting the Best Causal Inference Method for a Given Dataset](https://mlsanity.com/paper/1711.00083/) (2017)
- [Undersmoothing Causal Estimators with Generative Trees](https://mlsanity.com/paper/2203.08570/) (2022)

## Built on by (most cited first, 6 of 6)

- [Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimation](https://mlsanity.com/paper/2305.15984/) (2023)
- [Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choice](https://mlsanity.com/paper/2310.18212/) (2023)
- [Double Machine Learning for Static Panel Models with Fixed Effects](https://mlsanity.com/paper/2312.08174/) (2023)
- [Transfer Learning for Causal Effect Estimation](https://mlsanity.com/paper/2305.09126/) (2023)
- [Learned Causal Method Prediction](https://mlsanity.com/paper/2311.03989/) (2023)
- [Assessing Electricity Service Unfairness with Transfer Counterfactual Learning](https://mlsanity.com/paper/2310.03258/) (2023)

## Abstract (arXiv)

ML is playing an increasingly crucial role in estimating causal effects of treatments on outcomes from observational data. Many ML methods (`causal estimators') have been proposed for this task. All of these methods, as with any ML approach, require extensive hyperparameter tuning. For non-causal predictive tasks, there is a consensus on the choice of tuning metrics (e.g. mean squared error), making it simple to compare models. However, for causal inference tasks, such a consensus is yet to be reached, making any comparison of causal models difficult. On top of that, there is no ideal metric on which to tune causal estimators, so one must rely on proxies. Furthermore, the fact that model selection in causal inference involves multiple components (causal estimator, ML regressor, hyperparameters, metric), complicates the issue even further. In order to evaluate the importance of each component, we perform an extensive empirical study on their combination. Our experimental setup involves many commonly used causal estimators, regressors (`base learners' henceforth) and metrics applied to four well-known causal inference benchmark datasets. Our results show that hyperparameter tuning increased the probability of reaching state-of-the-art performance in average (65% rightarrow 81%) and individualised (50% rightarrow 57%) effect estimation with only commonly used estimators. We also show that the performance of standard metrics can be inconsistent across different scenarios. Our findings highlight the need for further research to establish whether metrics uniformly capable of state-of-the-art performance in causal model evaluation can be found.