# RealCause: Realistic Causal Inference Benchmarking

arXiv: https://arxiv.org/abs/2011.15007 · HTML: https://mlsanity.com/paper/2011.15007/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 30 of 46)

- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [Do-PFN: In-Context Learning for Causal Effect Estimation](https://mlsanity.com/paper/2506.06039/) (2025)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Causal Effect Inference for Structured Treatments](https://mlsanity.com/paper/2106.01939/) (2021)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data](https://mlsanity.com/paper/2210.17283/) (2022)
- [CausalRivers -- Scaling up benchmarking of causal discovery for real-world time-series](https://mlsanity.com/paper/2503.17452/) (2025)
- [Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.01939/) (2022)
- [DoWhy: Addressing Challenges in Expressing and Validating Causal Assumptions](https://mlsanity.com/paper/2108.13518/) (2021)
- [Validating Causal Inference Methods](https://mlsanity.com/paper/2202.04208/) (2022)
- [Membership Inference Attacks and Generalization: A Causal Perspective](https://mlsanity.com/paper/2209.08615/) (2022)
- [Counterfactual Generative Models for Time-Varying Treatments](https://mlsanity.com/paper/2305.15742/) (2023)
- [Towards Causal Foundation Model: on Duality between Causal Inference and Attention](https://mlsanity.com/paper/2310.00809/) (2023)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)
- [Marginal Causal Flows for Validation and Inference](https://mlsanity.com/paper/2411.01295/) (2024)
- [RCT Rejection Sampling for Causal Estimation Evaluation](https://mlsanity.com/paper/2307.15176/) (2023)
- [Use What You Know: Causal Foundation Models with Partial Graphs](https://mlsanity.com/paper/2602.14972/) (2026)
- [Quantitative probing: Validating causal models using quantitative domain knowledge](https://mlsanity.com/paper/2209.03013/) (2022)
- [The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation](https://mlsanity.com/paper/2303.01412/) (2023)
- [Learning When to Treat Business Processes: Prescriptive Process Monitoring with Causal Inference and Reinforcement Learning](https://mlsanity.com/paper/2303.03572/) (2023)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators](https://mlsanity.com/paper/2107.13346/) (2021)
- [Prescriptive Process Monitoring Under Resource Constraints: A Reinforcement Learning Approach](https://mlsanity.com/paper/2307.06564/) (2023)
- [Language Models as Causal Effect Generators](https://mlsanity.com/paper/2411.08019/) (2024)
- [SimBank: from Simulation to Solution in Prescriptive Process Monitoring](https://mlsanity.com/paper/2506.14772/) (2025)
- [Causal Foundation Models with Continuous Treatments](https://mlsanity.com/paper/2605.15133/) (2026)
- [Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation](https://mlsanity.com/paper/2307.04988/) (2023)
- [SpaCE: The Spatial Confounding Environment](https://mlsanity.com/paper/2312.00710/) (2023)
- [Undersmoothing Causal Estimators with Generative Trees](https://mlsanity.com/paper/2203.08570/) (2022)

## Abstract (arXiv)

There are many different causal effect estimators in causal inference. However, it is unclear how to choose between these estimators because there is no ground-truth for causal effects. A commonly used option is to simulate synthetic data, where the ground-truth is known. However, the best causal estimators on synthetic data are unlikely to be the best causal estimators on real data. An ideal benchmark for causal estimators would both (a) yield ground-truth values of the causal effects and (b) be representative of real data. Using flexible generative models, we provide a benchmark that both yields ground-truth and is realistic. Using this benchmark, we evaluate over 1500 different causal estimators and provide evidence that it is rational to choose hyperparameters for causal estimators using predictive metrics.