# Language Models as Causal Effect Generators

arXiv: https://arxiv.org/abs/2411.08019 · HTML: https://mlsanity.com/paper/2411.08019/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 68 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 28 of 28)

- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Conformal Inference of Counterfactuals and Individual Treatment Effects](https://mlsanity.com/paper/2006.06138/) (2020)
- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms](https://mlsanity.com/paper/2101.10943/) (2021)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [RealCause: Realistic Causal Inference Benchmarking](https://mlsanity.com/paper/2011.15007/) (2020)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [Prompting Large Language Models for Counterfactual Generation: An Empirical Study](https://mlsanity.com/paper/2305.14791/) (2023)
- [The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data](https://mlsanity.com/paper/1910.05387/) (2019)
- [Orthogonal Machine Learning: Power and Limitations](https://mlsanity.com/paper/1711.00342/) (2017)
- [CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning](https://mlsanity.com/paper/2106.13364/) (2021)
- [A Survey on Natural Language Counterfactual Generation](https://mlsanity.com/paper/2407.03993/) (2024)
- [How and Why to Use Experimental Data to Evaluate Methods for Observational Causal Inference](https://mlsanity.com/paper/2010.03051/) (2020)
- [Flexible text generation for counterfactual fairness probing](https://mlsanity.com/paper/2206.13757/) (2022)
- [Synth-Validation: Selecting the Best Causal Inference Method for a Given Dataset](https://mlsanity.com/paper/1711.00083/) (2017)
- [Sketch and Customize: A Counterfactual Story Generator](https://mlsanity.com/paper/2104.00929/) (2021)
- [On the Distinction Between "Conditional Average Treatment Effects" (CATE) and "Individual Treatment Effects" (ITE) Under Ignorability Assumptions](https://mlsanity.com/paper/2108.04939/) (2021)
- [Deep Autoregressive Models as Causal Inference Engines](https://mlsanity.com/paper/2409.18581/) (2024)

## Built on by (most cited first, 4 of 4)

- [Black Box Causal Inference: Effect Estimation via Meta Prediction](https://mlsanity.com/paper/2503.05985/) (2025)
- [PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimation](https://mlsanity.com/paper/2504.01509/) (2025)
- [Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Models](https://mlsanity.com/paper/2602.10282/) (2026)
- [CausalDS: Benchmarking Causal Reasoning in Data-Science Agents](https://mlsanity.com/paper/2607.08093/) (2026)

## Abstract (arXiv)

In this work, we present sequence-driven structural causal models (SD-SCMs), a framework for specifying causal models with user-defined structure and language-model-defined mechanisms. We characterize how an SD-SCM enables sampling from observational, interventional, and counterfactual distributions according to the desired causal structure. We then leverage this procedure to propose a new type of benchmark for causal inference methods, generating individual-level counterfactual data to test treatment effect estimation. We create an example benchmark consisting of thousands of datasets, and test a suite of popular estimation methods for average, conditional average, and individual treatment effect estimation. We find under this benchmark that (1) causal methods outperform non-causal methods and that (2) even state-of-the-art methods struggle with individualized effect estimation, suggesting this benchmark captures some inherent difficulties in causal estimation. Apart from generating data, this same technique can underpin the auditing of language models for (un)desirable causal effects, such as misinformation or discrimination. We believe SD-SCMs can serve as a useful tool in any application that would benefit from sequential data with controllable causal structure.