# Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption

arXiv: https://arxiv.org/abs/2508.08883 · HTML: https://mlsanity.com/paper/2508.08883/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 89 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 30 of 42)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game](https://mlsanity.com/paper/2102.13647/) (2021)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition](https://mlsanity.com/paper/1707.02641/) (2017)
- [Deep End-to-end Causal Inference](https://mlsanity.com/paper/2202.02195/) (2022)
- [Causal machine learning for predicting treatment outcomes](https://mlsanity.com/paper/2410.08770/) (2024)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [Causal normalizing flows: from theory to practice](https://mlsanity.com/paper/2306.05415/) (2023)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [Re-evaluating Evaluation](https://mlsanity.com/paper/1806.02643/) (2018)
- [RealCause: Realistic Causal Inference Benchmarking](https://mlsanity.com/paper/2011.15007/) (2020)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis](https://mlsanity.com/paper/1802.05046/) (2018)
- [Using Wasserstein Generative Adversarial Networks for the Design of Monte Carlo Simulations](https://mlsanity.com/paper/1909.02210/) (2019)
- [The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data](https://mlsanity.com/paper/1910.05387/) (2019)
- [CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data](https://mlsanity.com/paper/2210.17283/) (2022)
- [In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2302.02923/) (2023)
- [Assumption violations in causal discovery and the robustness of score matching](https://mlsanity.com/paper/2310.13387/) (2023)
- [Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.01939/) (2022)
- [Evaluation Methods and Measures for Causal Learning Algorithms](https://mlsanity.com/paper/2202.02896/) (2022)
- [Validating Causal Inference Methods](https://mlsanity.com/paper/2202.04208/) (2022)
- [Probabilistic Reasoning across the Causal Hierarchy](https://mlsanity.com/paper/2001.02889/) (2020)
- [textttcausalAssembly: Generating Realistic Production Data for Benchmarking Causal Discovery](https://mlsanity.com/paper/2306.10816/) (2023)
- [Counterfactual (Non-)identifiability of Learned Structural Causal Models](https://mlsanity.com/paper/2301.09031/) (2023)
- [Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability](https://mlsanity.com/paper/2206.08363/) (2022)
- [How and Why to Use Experimental Data to Evaluate Methods for Observational Causal Inference](https://mlsanity.com/paper/2010.03051/) (2020)
- [Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study](https://mlsanity.com/paper/2402.04674/) (2024)
- [Comparison of meta-learners for estimating multi-valued treatment heterogeneous effects](https://mlsanity.com/paper/2205.14714/) (2022)
- [Benchmark Data Repositories for Better Benchmarking](https://mlsanity.com/paper/2410.24100/) (2024)

## Built on by (most cited first, 6 of 6)

- [Overlap-weighted orthogonal meta-learner for treatment effect estimation over time](https://mlsanity.com/paper/2510.19643/) (2025)
- [DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series](https://mlsanity.com/paper/2607.27263/) (2026)
- [Toward Scalable and Valid Conditional Independence Testing with Spectral Representations](https://mlsanity.com/paper/2512.19510/) (2025)
- [Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation](https://mlsanity.com/paper/2605.10430/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)
- [CIDER-FM: Foundation Models for Causal Inference from Diverse Experimental Regimes](https://mlsanity.com/paper/2609.39523/) (2026)

## Abstract (arXiv)

Causal machine learning has the potential to revolutionize decision-making by combining the predictive power of machine learning algorithms with the theory of causal inference. However, these methods remain underutilized by the broader machine learning community, in part because current empirical evaluations do not permit assessment of their reliability and robustness, undermining their practical utility. Specifically, one of the principal criticisms made by the community is the extensive use of synthetic experiments. We argue, on the contrary, that synthetic experiments are essential and necessary to precisely assess and understand the capabilities of causal machine learning methods. To substantiate our position, we critically review the current evaluation practices, spotlight their shortcomings, and propose a set of principles for conducting rigorous empirical analyses with synthetic data. Adopting the proposed principles will enable comprehensive evaluations that build trust in causal machine learning methods, driving their broader adoption and impactful real-world use.