mlsanityData to 7 Oct 2026

Emergent Linear Representations in World Models of Self-Supervised Sequence Models

arXiv 2309.00941 ↗First posted 2 Sep 2023cs.LGcited by 313

On the map: World models → World models for language agents

Abstract (arXiv)

How do sequence models represent their decision-making process? Prior work suggests that Othello-playing neural network learned nonlinear models of the board state (Li et al., 2023). In this work, we provide evidence of a closely related linear representation of the board. In particular, we show that probing for "my colour" vs. "opponent's colour" may be a simple yet powerful way to interpret the model's internal state. This precise understanding of the internal representations allows us to control the model's behaviour with simple vector arithmetic. Linear representations enable significant interpretability progress, which we demonstrate with further exploration of how the world model is computed.

Key facts. Cited by 313 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 36 of its 55 references that are in the map. First posted on arXiv 2 Sep 2023.

Builds on

36 of 55 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2022Mass-Editing Memory in a Transformercited by 976
  10. 2022Toy Models of Superpositioncited by 928
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
24 more
  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  4. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  5. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  7. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  8. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  9. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  11. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  12. 2020Compositional Explanations of Neuronscited by 172
  13. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  14. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  15. 2022Linearly Mapping from Image to Text Spacecited by 139
  16. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  17. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  18. 2017Residual Connections Encourage Iterative Inferencecited by 74
  19. 2022Polysemanticity and Capacity in Neural Networkscited by 74
  20. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61
  21. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  22. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  23. 2022Probing for Incremental Parse States in Autoregressive Language Modelscited by 24
  24. 2022Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hexcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 83; 2025: 113; 2026: 102.

313 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  6. 2023Language Models Represent Space and Timecited by 306
  7. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  8. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  9. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  10. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  11. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  12. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
38 more
  1. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  2. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  3. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  4. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  5. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  6. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  7. 2025Model Organisms for Emergent Misalignmentcited by 84
  8. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  9. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  10. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  11. 2024Transformers represent belief state geometry in their residual streamcited by 73
  12. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  13. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
  14. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  15. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 64
  16. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 60
  17. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 57
  18. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  19. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Modelscited by 51
  20. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  21. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  22. 2024ICLR: In-Context Learning of Representationscited by 49
  23. 2023From task structures to world models: What do LLMs know?cited by 46
  24. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  25. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 46
  26. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  27. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
  28. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  29. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  30. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  31. 2024Decomposing The Dark Matter of Sparse Autoencoderscited by 42
  32. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzlescited by 35
  33. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  34. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  35. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  36. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 33
  37. 2025(How) Do Language Models Track State?cited by 31
  38. 2024Standards for Belief Representations in LLMscited by 30

50 most cited of 313 All 313 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT
  2. 2026Platonic Representation Hypothesis on World Models
  3. 2024Disentangling Representations through Multi-task Learning
  4. 2025How GPT learns layer by layer
  5. 2024State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?
  6. 2026Transformers Linearly Represent Highly Structured World Models
  7. 2025Assessing Adaptive World Models in Machines with Novel Games
  8. 2026The Linear Representation Hypothesis for Vision-Language-Action Models