# Words as Difference Makers: How Large Language Models Determine Causal Structure in Text

arXiv: https://arxiv.org/abs/2606.22430 · HTML: https://mlsanity.com/paper/2606.22430/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 2 of its 23 references that are in the map.
- First posted on arXiv 21 Jun 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 2 of 2)

- ["Understanding AI": Semantic Grounding in Large Language Models](https://mlsanity.com/paper/2402.10992/) (2024)
- [Machine Learning and Theory Ladenness -- A Phenomenological Account](https://mlsanity.com/paper/2409.11277/) (2024)

## Abstract (arXiv)

Because large language models (LLMs) are impressively successful in predicting text, it appears that they must have access to a 'world model' representing causal and definitional structure. However, the dominant formalisms of modern causal inference -- Judea Pearl's interventionist approach and the Neyman-Rubin potential outcomes framework -- struggle to illuminate how LLMs learn causal structure. I resolve this puzzle by arguing that LLMs employ a specific inductive approach based on a difference-making logic -- sometimes called variational induction. I demonstrate how central aspects of this logic are realized during training, where LLMs require enormous amounts of text data from a wide range of contexts to identify difference- and indifference-makers within word sequences. Furthermore, I analyze specific architectural features of LLMs -- such as token embeddings and self-attention -- to determine their roles in variational induction. The difference-making logic of LLMs fundamentally parallels the experimental method, where causal relations are derived by systematically varying individual circumstances to determine their influence on a phenomenon.