# Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis

arXiv: https://arxiv.org/abs/1802.05046 · HTML: https://mlsanity.com/paper/1802.05046/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2018.

## Built on by (most cited first, 30 of 33)

- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data](https://mlsanity.com/paper/2210.17283/) (2022)
- [CausalRivers -- Scaling up benchmarking of causal discovery for real-world time-series](https://mlsanity.com/paper/2503.17452/) (2025)
- [Estimating Average Treatment Effects via Orthogonal Regularization](https://mlsanity.com/paper/2101.08490/) (2021)
- [Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.01939/) (2022)
- [Evaluation Methods and Measures for Causal Learning Algorithms](https://mlsanity.com/paper/2202.02896/) (2022)
- [Normalizing Flows for Interventional Density Estimation](https://mlsanity.com/paper/2209.06203/) (2022)
- [Causal Evaluation of Language Models](https://mlsanity.com/paper/2405.00622/) (2024)
- [Zero-shot causal learning](https://mlsanity.com/paper/2301.12292/) (2023)
- [Towards Causal Foundation Model: on Duality between Causal Inference and Attention](https://mlsanity.com/paper/2310.00809/) (2023)
- [A Survey of Deep Causal Models and Their Industrial Applications](https://mlsanity.com/paper/2209.08860/) (2022)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)
- [RCT Rejection Sampling for Causal Estimation Evaluation](https://mlsanity.com/paper/2307.15176/) (2023)
- [Proximal Causal Inference With Text Data](https://mlsanity.com/paper/2401.06687/) (2024)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Bayesian Model Averaging for Causality Estimation and its Approximation based on Gaussian Scale Mixture Distributions](https://mlsanity.com/paper/2103.08195/) (2021)
- [Causal Effect Estimation using Variational Information Bottleneck](https://mlsanity.com/paper/2110.13705/) (2021)
- [Uncertainty Quantification in Heterogeneous Treatment Effect Estimation with Gaussian-Process-Based Partially Linear Model](https://mlsanity.com/paper/2312.10435/) (2023)
- [CausalMan: A physics-based simulator for large-scale causality](https://mlsanity.com/paper/2502.12707/) (2025)
- [Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation](https://mlsanity.com/paper/2307.04988/) (2023)
- [Progressive Generalization Risk Reduction for Data-Efficient Causal Effect Estimation](https://mlsanity.com/paper/2411.11256/) (2024)
- [Enhancing Treatment Effect Estimation via Active Learning: A Counterfactual Covering Perspective](https://mlsanity.com/paper/2505.05242/) (2025)
- [Flow-based Generative Modeling of Potential Outcomes and Counterfactuals](https://mlsanity.com/paper/2505.16051/) (2025)
- [How to select predictive models for causal inference?](https://mlsanity.com/paper/2302.00370/) (2023)
- [Differentiable Pareto-Smoothed Weighting for High-Dimensional Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2404.17483/) (2024)
- [Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community](https://mlsanity.com/paper/2110.07938/) (2021)
- [Integrating Nearest Neighbors with Neural Network Models for Treatment Effect Estimation](https://mlsanity.com/paper/2305.06789/) (2023)
- [ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects](https://mlsanity.com/paper/2111.06811/) (2021)
- [C-XGBoost: A tree boosting model for causal effect estimation](https://mlsanity.com/paper/2404.00751/) (2024)

## Abstract (arXiv)

Causal inference analysis is the estimation of the effects of actions on outcomes. In the context of healthcare data this means estimating the outcome of counter-factual treatments (i.e. including treatments that were not observed) on a patient's outcome. Compared to classic machine learning methods, evaluation and validation of causal inference analysis is more challenging because ground truth data of counter-factual outcome can never be obtained in any real-world scenario. Here, we present a comprehensive framework for benchmarking algorithms that estimate causal effect. The framework includes unlabeled data for prediction, labeled data for validation, and code for automatic evaluation of algorithm predictions using both established and novel metrics. The data is based on real-world covariates, and the treatment assignments and outcomes are based on simulations, which provides the basis for validation. In this framework we address two questions: one of scaling, and the other of data-censoring. The framework is available as open source code at https://github.com/IBM-HRL-MLHLS/IBM-Causal-Inference-Benchmarking-Framework