# MetaOthello: A Controlled Study of Multiple World Models in Transformers

arXiv: https://arxiv.org/abs/2602.23164 · HTML: https://mlsanity.com/paper/2602.23164/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 17 of 17)

- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models](https://mlsanity.com/paper/2403.15498/) (2024)
- [A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates](https://mlsanity.com/paper/2401.03910/) (2024)
- [mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?](https://mlsanity.com/paper/2404.12444/) (2024)
- [When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks](https://mlsanity.com/paper/2210.12786/) (2022)
- [Revisiting the Othello World Model Hypothesis](https://mlsanity.com/paper/2503.04421/) (2025)
- [What Does it Mean for a Neural Network to Learn a "World Model"?](https://mlsanity.com/paper/2507.21513/) (2025)
- [A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment](https://mlsanity.com/paper/2412.07446/) (2024)

## Built on by (most cited first, 4 of 4)

- [When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks](https://mlsanity.com/paper/2604.27272/) (2026)
- [A Pre-Specified Construction-Confirmation Test of Operation-Level Causal Transfer Across Finite Isomorphic Symbolic Domains](https://mlsanity.com/paper/2608.15809/) (2026)
- [Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs](https://mlsanity.com/paper/2609.24352/) (2026)
- [When Generative Models Act as Simulators](https://mlsanity.com/paper/2610.12847/) (2026)

## Abstract (arXiv)

Foundation models must handle multiple generative processes, yet mechanistic interpretability largely studies capabilities in isolation; it remains unclear how a single transformer organizes multiple, potentially conflicting "world models". Previous experiments on Othello playing neural-networks test world-model learning but focus on a single game with a single set of rules. We introduce MetaOthello, a controlled suite of Othello variants with shared syntax but different rules or tokenizations, and train small GPTs on mixed-variant data to study how multiple world models are organized in a shared representation space. We find that transformers trained on mixed-game data do not partition their capacity into isolated sub-models; instead, they converge on a mostly shared board-state representation that transfers causally across variants. Linear probes trained on one variant can intervene on another's internal state with effectiveness approaching that of matched probes. For isomorphic games with token remapping, representations are equivalent up to a single orthogonal rotation that generalizes across layers. When rules partially overlap, early layers maintain game-agnostic representations while a middle layer identifies game identity, and later layers specialize. MetaOthello offers a path toward understanding not just whether transformers learn world models, but how they organize many at once.