mlsanityData to 14 Oct 2026

Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT

arXiv 2310.07582 ↗First posted 11 Oct 2023cs.LG cs.AIcited by 16

On the map: World models → World models for language agents

Abstract (arXiv)

Foundation models exhibit significant capabilities in decision-making and logical deductions. Nonetheless, a continuing discourse persists regarding their genuine understanding of the world as opposed to mere stochastic mimicry. This paper meticulously examines a simple transformer trained for Othello, extending prior research to enhance comprehension of the emergent world model of Othello-GPT. The investigation reveals that Othello-GPT encapsulates a linear representation of opposing pieces, a factor that causally steers its decision-making process. This paper further elucidates the interplay between the linear world representation and causal decision-making, and their dependence on layer depth and model complexity. We have made the code public.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 8 of its 14 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

8 of 14 references.

References · 8

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  6. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  7. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  8. 2021Deep ViT Features as Dense Visual Descriptorscited by 330

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 9; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  2. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 47
  3. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 29
  4. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  5. 2025Revisiting the Othello World Model Hypothesiscited by 11
  6. 2025Linear Spatial World Models Emerge in Large Language Modelscited by 6
  7. 2023Emergence and Function of Abstract Representations in Self-Supervised Transformerscited by 3
  8. 2025How GPT learns layer by layercited by 2
  9. 2025Can LLMs Learn to Map the World from Local Descriptions?cited by 1
  10. 2025Internal Planning in Language Models: Characterizing Horizon and Branch Awarenesscited by 1
  11. 2025Representation in large language modelscited by 0
  12. 2025Causal Masking on Spatial Data: An Information-Theoretic Case for Learning Spatial Datasets with Unimodal Language Modelscited by 0
4 more
  1. 2025Automatically Finding Rule-Based Neurons in OthelloGPTcited by 0
  2. 2026A geometric relation of the error introduced by sampling a language model's output distribution to its internal statecited by 0
  3. 2026Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoningcited by 0
  4. 2026When Generative Models Act as Simulatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MetaOthello: A Controlled Study of Multiple World Models in Transformers
  2. 2026World Modeling in Transformers
  3. 2023A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoning
  4. 2024Transformers Use Causal World Models in Maze-Solving Tasks
  5. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning
  6. 2026Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking
  7. 2026Emergent Analogical Reasoning in Transformers
  8. 2026Transformers Linearly Represent Highly Structured World Models