mlsanityData to 14 Oct 2026

Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption

arXiv 2508.08883 ↗First posted 12 Aug 2025cs.LG cs.AI stat.MEcited by 6

Not on the map. Why?

Abstract (arXiv)

Causal machine learning has the potential to revolutionize decision-making by combining the predictive power of machine learning algorithms with the theory of causal inference. However, these methods remain underutilized by the broader machine learning community, in part because current empirical evaluations do not permit assessment of their reliability and robustness, undermining their practical utility. Specifically, one of the principal criticisms made by the community is the extensive use of synthetic experiments. We argue, on the contrary, that synthetic experiments are essential and necessary to precisely assess and understand the capabilities of causal machine learning methods. To substantiate our position, we critically review the current evaluation practices, spotlight their shortcomings, and propose a set of principles for conducting rigorous empirical analyses with synthetic data. Adopting the proposed principles will enable comprehensive evaluations that build trust in causal machine learning methods, driving their broader adoption and impactful real-world use.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 42 of its 89 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

42 of 89 references.

References · 42

  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2021Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Gamecited by 161
  4. 2021The Benchmark Lotterycited by 136
  5. 2017Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competitioncited by 106
  6. 2022Deep End-to-end Causal Inferencecited by 95
  7. 2024Causal machine learning for predicting treatment outcomescited by 74
  8. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  9. 2023Causal normalizing flows: from theory to practicecited by 51
  10. 2022Evaluation Gaps in Machine Learning Practicecited by 49
  11. 2018Re-evaluating Evaluationcited by 47
  12. 2020RealCause: Realistic Causal Inference Benchmarkingcited by 46
30 more
  1. 2021How to avoid machine learning pitfalls: a guide for academic researcherscited by 35
  2. 2018Benchmarking Framework for Performance-Evaluation of Causal Inference Analysiscited by 33
  3. 2019Using Wasserstein Generative Adversarial Networks for the Design of Monte Carlo Simulationscited by 32
  4. 2019The Case for Evaluating Causal Models Using Interventional Measures and Empirical Datacited by 30
  5. 2022CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Datacited by 29
  6. 2023In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimationcited by 29
  7. 2023Assumption violations in causal discovery and the robustness of score matchingcited by 28
  8. 2022Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimationcited by 25
  9. 2022Evaluation Methods and Measures for Causal Learning Algorithmscited by 23
  10. 2022Validating Causal Inference Methodscited by 20
  11. 2020Probabilistic Reasoning across the Causal Hierarchycited by 19
  12. 2023textttcausalAssembly: Generating Realistic Production Data for Benchmarking Causal Discoverycited by 18
  13. 2023Counterfactual (Non-)identifiability of Learned Structural Causal Modelscited by 17
  14. 2022Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretabilitycited by 15
  15. 2020How and Why to Use Experimental Data to Evaluate Methods for Observational Causal Inferencecited by 13
  16. 2024Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Studycited by 11
  17. 2022Comparison of meta-learners for estimating multi-valued treatment heterogeneous effectscited by 10
  18. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  19. 2024Learning Structural Causal Models through Deep Generative Models: Methods, Guarantees, and Challengescited by 8
  20. 2024Marginal Causal Flows for Validation and Inferencecited by 8
  21. 2023RCT Rejection Sampling for Causal Estimation Evaluationcited by 7
  22. 2022All models are wrong, but which are useful? Comparing parametric and nonparametric estimation of causal effects in finite samplescited by 6
  23. 2022Perfect Sampling from Pairwise Comparisonscited by 6
  24. 2024Position: Embracing Negative Results in Machine Learningcited by 6
  25. 2024Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithmscited by 5
  26. 2023Prescriptive Process Monitoring Under Resource Constraints: A Reinforcement Learning Approachcited by 4
  27. 2023Learned Causal Method Predictioncited by 3
  28. 2024Position: The Causal Revolution Needs Scientific Pragmatismcited by 3
  29. 2024Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimatorscited by 2
  30. 2024Hierarchical Bias-Driven Stratification for Interpretable Causal Effect Estimationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Overlap-weighted orthogonal meta-learner for treatment effect estimation over timecited by 3
  2. 2026DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Seriescited by 2
  3. 2025Toward Scalable and Valid Conditional Independence Testing with Spectral Representationscited by 1
  4. 2026Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimationcited by 0
  5. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
  6. 2026CIDER-FM: Foundation Models for Causal Inference from Diverse Experimental Regimescited by 0