# Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

arXiv: https://arxiv.org/abs/2009.09961 · HTML: https://mlsanity.com/paper/2009.09961/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 13 of 13)

- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [A Causal View of Entity Bias in (Large) Language Models](https://mlsanity.com/paper/2305.14695/) (2023)
- [Zero-shot causal learning](https://mlsanity.com/paper/2301.12292/) (2023)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)
- [RCT Rejection Sampling for Causal Estimation Evaluation](https://mlsanity.com/paper/2307.15176/) (2023)
- [Stranger Danger! Cross-Community Interactions with Fringe Users Increase the Growth of Fringe Communities on Reddit](https://mlsanity.com/paper/2310.12186/) (2023)
- [Causal Inference from Text: Unveiling Interactions between Variables](https://mlsanity.com/paper/2311.05286/) (2023)
- [Estimating Causal Effects of Text Interventions Leveraging LLMs](https://mlsanity.com/paper/2410.21474/) (2024)
- [LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding](https://mlsanity.com/paper/2507.02843/) (2025)
- [Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community](https://mlsanity.com/paper/2110.07938/) (2021)
- [LLM-Extracted Covariates for Clinical Causal Inference: Rethinking Integration Strategies](https://mlsanity.com/paper/2604.16763/) (2026)
- [Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings](https://mlsanity.com/paper/2606.04286/) (2026)
- [From Web(logs) to Web(AI): Questions, Platforms, and Methods across Twenty Editions of ICWSM](https://mlsanity.com/paper/2610.00205/) (2026)

## Abstract (arXiv)

Causal inference studies using textual social media data can provide actionable insights on human behavior. Making accurate causal inferences with text requires controlling for confounding which could otherwise impart bias. Recently, many different methods for adjusting for confounders have been proposed, and we show that these existing methods disagree with one another on two datasets inspired by previous social media studies. Evaluating causal methods is challenging, as ground truth counterfactuals are almost never available. Presently, no empirical evaluation framework for causal methods using text exists, and as such, practitioners must select their methods without guidance. We contribute the first such framework, which consists of five tasks drawn from real world studies. Our framework enables the evaluation of any casual inference method using text. Across 648 experiments and two datasets, we evaluate every commonly used causal inference method and identify their strengths and weaknesses to inform social media researchers seeking to use such methods, and guide future improvements. We make all tasks, data, and models public to inform applications and encourage additional research.