# Conceptualizing Treatment Leakage in Text-based Causal Inference

arXiv: https://arxiv.org/abs/2205.00465 · HTML: https://mlsanity.com/paper/2205.00465/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 1 May 2022.

## Builds on (most cited first, 11 of 11)

- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates](https://mlsanity.com/paper/2005.00649/) (2020)
- [A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Models](https://mlsanity.com/paper/2102.06648/) (2021)
- [Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality](https://mlsanity.com/paper/1801.00644/) (2018)
- [Statistical modeling: the three cultures](https://mlsanity.com/paper/2012.04570/) (2020)
- [Personalized Public Policy Analysis in Social Sciences using Causal-Graphical Normalizing Flows](https://mlsanity.com/paper/2202.03281/) (2022)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)
- [On the Monotonicity of a Nondifferentially Mismeasured Binary Confounder](https://mlsanity.com/paper/2005.13245/) (2020)

## Built on by (most cited first, 14 of 14)

- [Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatments](https://mlsanity.com/paper/2410.00903/) (2024)
- [Integrating Earth Observation Data into Causal Inference: Challenges and Opportunities](https://mlsanity.com/paper/2301.12985/) (2023)
- [Benchmarking Debiasing Methods for LLM-based Parameter Estimates](https://mlsanity.com/paper/2506.09627/) (2025)
- [A Scoping Review of Earth Observation and Machine Learning for Causal Inference: Implications for the Geography of Poverty](https://mlsanity.com/paper/2406.02584/) (2024)
- [Can Large Language Models (or Humans) Disentangle Text?](https://mlsanity.com/paper/2403.16584/) (2024)
- [LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding](https://mlsanity.com/paper/2507.02843/) (2025)
- [Leveraging Generative Artificial Intelligence for Causal Inference with Unstructured Data](https://mlsanity.com/paper/2507.03897/) (2025)
- [Debiasing Machine Learning Predictions for Causal Inference Without Additional Ground Truth Data: "One Map, Many Trials" in Satellite-Driven Poverty Analysis](https://mlsanity.com/paper/2508.01341/) (2025)
- [Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?](https://mlsanity.com/paper/2508.01109/) (2025)
- [What Happens to a Dataset Transformed by a Projection-based Concept Removal Method?](https://mlsanity.com/paper/2403.16142/) (2024)
- [Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Records](https://mlsanity.com/paper/2507.20993/) (2025)
- [Detecting and Mitigating Treatment Leakage in Text-Based Causal Inference: Distillation and Sensitivity Analysis](https://mlsanity.com/paper/2601.02400/) (2025)
- [When AI Generates Covariates: Causal Typing and Estimand Drift in Sequential Experiments](https://mlsanity.com/paper/2609.17772/) (2026)
- [Information Set Emulation: Causal Certificates for AI Derived EHR Features](https://mlsanity.com/paper/2609.17777/) (2026)

## Abstract (arXiv)

Causal inference methods that control for text-based confounders are becoming increasingly important in the social sciences and other disciplines where text is readily available. However, these methods rely on a critical assumption that there is no treatment leakage: that is, the text only contains information about the confounder and no information about treatment assignment. When this assumption does not hold, methods that control for text to adjust for confounders face the problem of post-treatment (collider) bias. However, the assumption that there is no treatment leakage may be unrealistic in real-world situations involving text, as human language is rich and flexible. Language appearing in a public policy document or health records may refer to the future and the past simultaneously, and thereby reveal information about the treatment assignment. In this article, we define the treatment-leakage problem, and discuss the identification as well as the estimation challenges it raises. Second, we delineate the conditions under which leakage can be addressed by removing the treatment-related signal from the text in a pre-processing step we define as text distillation. Lastly, using simulation, we show how treatment leakage introduces a bias in estimates of the average treatment effect (ATE) and how text distillation can mitigate this bias.