# IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark

arXiv: https://arxiv.org/abs/2405.16069 · HTML: https://mlsanity.com/paper/2405.16069/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 15 of 15)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Fair Inference On Outcomes](https://mlsanity.com/paper/1705.10378/) (2017)
- [Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers](https://mlsanity.com/paper/1912.03277/) (2019)
- [Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition](https://mlsanity.com/paper/1707.02641/) (2017)
- [DoWhy: An End-to-End Library for Causal Inference](https://mlsanity.com/paper/2011.04216/) (2020)
- [CausalML: Python Package for Causal Machine Learning](https://mlsanity.com/paper/2002.11631/) (2020)
- [A causal framework for discovering and removing direct and indirect discrimination](https://mlsanity.com/paper/1611.07509/) (2016)
- [On the Fairness of Causal Algorithmic Recourse](https://mlsanity.com/paper/2010.06529/) (2020)
- [The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data](https://mlsanity.com/paper/1910.05387/) (2019)
- [Atlantic Causal Inference Conference (ACIC) Data Analysis Challenge 2017](https://mlsanity.com/paper/1905.09515/) (2019)
- [Causal Discovery for Fairness](https://mlsanity.com/paper/2206.06685/) (2022)
- [The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation](https://mlsanity.com/paper/2106.04240/) (2021)
- [Causal Decision Trees](https://mlsanity.com/paper/1508.03812/) (2015)

## Built on by (most cited first, 1 of 1)

- [CAF'E: Causal Black-Box Testing of Machine Unlearning](https://mlsanity.com/paper/2509.16525/) (2025)

## Abstract (arXiv)

Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.