# Detecting and Mitigating Treatment Leakage in Text-Based Causal Inference: Distillation and Sensitivity Analysis

arXiv: https://arxiv.org/abs/2601.02400 · HTML: https://mlsanity.com/paper/2601.02400/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 30 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [An Introduction to Proximal Causal Learning](https://mlsanity.com/paper/2009.10982/) (2020)
- [Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates](https://mlsanity.com/paper/2005.00649/) (2020)
- [Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models](https://mlsanity.com/paper/2306.04746/) (2023)
- [Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality](https://mlsanity.com/paper/1801.00644/) (2018)
- [Image-based Treatment Effect Heterogeneity](https://mlsanity.com/paper/2206.06417/) (2022)
- [Estimating Causal Effects of Tone in Online Debates](https://mlsanity.com/paper/1906.04177/) (2019)
- [Statistical modeling: the three cultures](https://mlsanity.com/paper/2012.04570/) (2020)
- [Conceptualizing Treatment Leakage in Text-based Causal Inference](https://mlsanity.com/paper/2205.00465/) (2022)
- [Deep Learning With DAGs](https://mlsanity.com/paper/2401.06864/) (2024)
- [A Scoping Review of Earth Observation and Machine Learning for Causal Inference: Implications for the Geography of Poverty](https://mlsanity.com/paper/2406.02584/) (2024)
- [Can Large Language Models (or Humans) Disentangle Text?](https://mlsanity.com/paper/2403.16584/) (2024)
- [Leveraging Compact Satellite Embeddings and Graph Neural Networks for Large-Scale Poverty Mapping](https://mlsanity.com/paper/2511.01408/) (2025)
- [Measuring poverty in India with machine learning and remote sensing](https://mlsanity.com/paper/2202.00109/) (2021)
- [Debiasing Machine Learning Predictions for Causal Inference Without Additional Ground Truth Data: "One Map, Many Trials" in Satellite-Driven Poverty Analysis](https://mlsanity.com/paper/2508.01341/) (2025)

## Abstract (arXiv)

Text-based causal inference increasingly employs textual data as proxies for unobserved confounders, yet this approach introduces a previously undertheorized source of bias: treatment leakage. Treatment leakage occurs when text intended to capture confounding information also contains signals predictive of treatment status, thereby inducing post-treatment bias in causal estimates. Critically, this problem can arise even when documents precede treatment assignment, as authors may employ future-referencing language that anticipates subsequent interventions. Despite growing recognition of this issue, no systematic methods exist for identifying and mitigating treatment leakage in text-as-confounder applications. This paper addresses this gap through three contributions. First, we provide formal statistical and set-theoretic definitions of treatment leakage that clarify when and why bias occurs. Second, we propose four text distillation methods -- similarity-based passage removal, distant supervision classification, salient feature removal, and iterative nullspace projection -- designed to eliminate treatment-predictive content while preserving confounder information. Third, we validate these methods through simulations using synthetic text and an empirical application examining International Monetary Fund structural adjustment programs and child mortality. Our findings indicate that moderate distillation optimally balances bias reduction against confounder retention, whereas overly stringent approaches degrade estimate precision.