# A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment

arXiv: https://arxiv.org/abs/2412.07446 · HTML: https://mlsanity.com/paper/2412.07446/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Chess as a Testbed for Language Model State Tracking](https://mlsanity.com/paper/2102.13249/) (2021)
- [Learning Sparse Causal Models is not NP-hard](https://mlsanity.com/paper/1309.6824/) (2013)
- [Causal Interpretation of Self-Attention in Pre-Trained Transformers](https://mlsanity.com/paper/2310.20307/) (2023)
- [Iterative Causal Discovery in the Possible Presence of Latent Confounders and Selection Bias](https://mlsanity.com/paper/2111.04095/) (2021)
- [Bayesian Structure Learning by Recursive Bootstrap](https://mlsanity.com/paper/1809.04828/) (2018)
- [Improving Efficiency and Accuracy of Causal Discovery Using a Hierarchical Wrapper](https://mlsanity.com/paper/2107.05001/) (2021)
- [CLEAR: Causal Explanations from Attention in Neural Recommenders](https://mlsanity.com/paper/2210.10621/) (2022)

## Built on by (most cited first, 6 of 6)

- [MetaOthello: A Controlled Study of Multiple World Models in Transformers](https://mlsanity.com/paper/2602.23164/) (2026)
- [Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process](https://mlsanity.com/paper/2509.17380/) (2025)
- [Can Neural Networks Learn Small Algebraic Worlds? An Investigation Into the Group-theoretic Structures Learned By Narrow Models Trained To Predict Group Operations](https://mlsanity.com/paper/2601.21150/) (2026)
- [From Next Token Prediction to (STRIPS) World Models](https://mlsanity.com/paper/2509.13389/) (2025)
- [MoRE: Batch-Robust Multi-Omics Representations from Frozen Pre-trained Transformers](https://mlsanity.com/paper/2511.20382/) (2025)
- [Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs](https://mlsanity.com/paper/2609.24352/) (2026)

## Abstract (arXiv)

Are generative pre-trained transformer (GPT) models, trained only to predict the next token, implicitly learning a world model from which sequences are generated one token at a time? We address this question by deriving a causal interpretation of the attention mechanism in GPT and presenting a causal world model that arises from this interpretation. Furthermore, we propose that GPT models, at inference time, can be utilized for zero-shot causal structure learning for input sequences, and introduce a corresponding confidence score. Empirical tests were conducted in controlled environments using the setups of the Othello and Chess strategy games. A GPT, pre-trained on real-world games played with the intention of winning, was tested on out-of-distribution synthetic data consisting of sequences of random legal moves. We find that the GPT model is likely to generate legal next moves for out-of-distribution sequences for which a causal structure is encoded in the attention mechanism with high confidence. In cases where it generates illegal moves, it also fails to capture a causal structure.