# Predicting Causal Effects from Natural Language Queries using Structured Representations

arXiv: https://arxiv.org/abs/2605.29631 · HTML: https://mlsanity.com/paper/2605.29631/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 14 of 14)

- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Finetuning LLMs for Human Behavior Prediction in Social Science Experiments](https://mlsanity.com/paper/2509.05830/) (2025)
- [Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatments](https://mlsanity.com/paper/2410.00903/) (2024)
- [Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs](https://mlsanity.com/paper/2305.03642/) (2023)
- [Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges](https://mlsanity.com/paper/2303.05392/) (2023)
- [Proximal Causal Inference With Text Data](https://mlsanity.com/paper/2401.06687/) (2024)
- [Contrastive representations of high-dimensional, structured treatments](https://mlsanity.com/paper/2411.19245/) (2024)
- [Controlling Learned Effects to Reduce Spurious Correlations in Text Classifiers](https://mlsanity.com/paper/2305.16863/) (2023)
- [M3Retrieve: Benchmarking Multimodal Retrieval for Medicine](https://mlsanity.com/paper/2510.06888/) (2025)
- [Gradient-based inference of abstract task representations for generalization in neural networks](https://mlsanity.com/paper/2407.17356/) (2024)
- [AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs](https://mlsanity.com/paper/2409.09704/) (2024)
- [Estimating Causal Effects of Text Interventions Leveraging LLMs](https://mlsanity.com/paper/2410.21474/) (2024)

## Abstract (arXiv)

Randomized controlled trials are a cornerstone of medicine and the social sciences as they enable reliable estimates of causal effects. However, they are costly and time-consuming to conduct, motivating interest in predicting causal effects from existing experimental evidence. Recent advances in large language models (LLMs) have demonstrated strong performance on knowledge-intensive tasks, raising the question of whether these models can be used for forecasting causal effect sizes. To investigate this, we introduce Query2Effect, a new large-scale benchmark consisting of more than 72,000 natural language questions aligned with experiment descriptions, created to simulate realistic information-seeking scenarios by varying query specificity along dimensions of implicitness, abstraction, and ambiguity. We then propose a two-step framework that first generates a synthetic structured representation of a query before predicting effect size using a supervised encoder model. Experiments show that finetuning plays a crucial role in improving prediction performance, with absolute error reducing by -27% up to -71% compared to prompted out-of-the-box LLMs, and that our two-step framework is beneficial for out-of-domain generalization, highlighting the benefits of separating semantic interpretation from numerical effect estimation.