# The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data

arXiv: https://arxiv.org/abs/1910.05387 · HTML: https://mlsanity.com/paper/1910.05387/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 30 of 30)

- [Large-Scale Differentiable Causal Discovery of Factor Graphs](https://mlsanity.com/paper/2206.07824/) (2022)
- [Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling](https://mlsanity.com/paper/2211.03553/) (2022)
- [CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data](https://mlsanity.com/paper/2210.17283/) (2022)
- [Evaluation Methods and Measures for Causal Learning Algorithms](https://mlsanity.com/paper/2202.02896/) (2022)
- [Validating Causal Inference Methods](https://mlsanity.com/paper/2202.04208/) (2022)
- [NODAGS-Flow: Nonlinear Cyclic Causal Structure Learning](https://mlsanity.com/paper/2301.01849/) (2023)
- [textttcausalAssembly: Generating Realistic Production Data for Benchmarking Causal Discovery](https://mlsanity.com/paper/2306.10816/) (2023)
- [The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applications](https://mlsanity.com/paper/2412.01953/) (2024)
- [Adjustment Identification Distance: A gadjid for Causal Structure Learning](https://mlsanity.com/paper/2402.08616/) (2024)
- [The Proximal ID Algorithm](https://mlsanity.com/paper/2108.06818/) (2021)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)
- [RCT Rejection Sampling for Causal Estimation Evaluation](https://mlsanity.com/paper/2307.15176/) (2023)
- [The CausalBench challenge: A machine learning contest for gene network inference from single-cell perturbation data](https://mlsanity.com/paper/2308.15395/) (2023)
- [Proximal Causal Inference With Text Data](https://mlsanity.com/paper/2401.06687/) (2024)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithms](https://mlsanity.com/paper/2412.10039/) (2024)
- [Reinterpreting causal discovery as the task of predicting unobserved joint statistics](https://mlsanity.com/paper/2305.06894/) (2023)
- [Language Models as Causal Effect Generators](https://mlsanity.com/paper/2411.08019/) (2024)
- [CausalMan: A physics-based simulator for large-scale causality](https://mlsanity.com/paper/2502.12707/) (2025)
- [Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community](https://mlsanity.com/paper/2110.07938/) (2021)
- [IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark](https://mlsanity.com/paper/2405.16069/) (2024)
- [Learning Personalized Treatment Decisions in Precision Medicine: Disentangling Treatment Assignment Bias in Counterfactual Outcome Prediction and Biomarker Identification](https://mlsanity.com/paper/2410.00509/) (2024)
- [Heterogeneous Causal Discovery of Repeated Undesirable Health Outcomes](https://mlsanity.com/paper/2503.11477/) (2025)
- [Effect-Level Validation for Causal Discovery](https://mlsanity.com/paper/2602.08340/) (2026)
- [Consistency evaluation of benchmarks used for causal discovery](https://mlsanity.com/paper/2606.01789/) (2026)
- [ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects](https://mlsanity.com/paper/2111.06811/) (2021)
- [A Data-Driven Two-Phase Multi-Split Causal Ensemble Model for Time Series](https://mlsanity.com/paper/2403.04793/) (2024)
- [Graph Distance Based on Cause-Effect Estimands with Latents](https://mlsanity.com/paper/2510.25037/) (2025)
- [A Topological Sorting Criterion for Random Causal Directed Acyclic Graphs](https://mlsanity.com/paper/2605.06288/) (2026)
- [ORCA: An End-to-End Interactive Copilot for Optimized Root Cause Analysis](https://mlsanity.com/paper/2605.27022/) (2026)

## Abstract (arXiv)

Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of impressive technical advances. However, evaluation techniques for causal modeling algorithms have remained somewhat primitive, limiting what we can learn from experimental studies of algorithm performance, constraining the types of algorithms and model representations that researchers consider, and creating a gap between theory and practice. We argue for more frequent use of evaluation techniques that examine interventional measures rather than structural or observational measures, and that evaluate those measures on empirical data rather than synthetic data. We survey the current practice in evaluation and show that the techniques we recommend are rarely used in practice. We show that such techniques are feasible and that data sets are available to conduct such evaluations. We also show that these techniques produce substantially different results than using structural measures and synthetic data.