mlsanityData to 7 Oct 2026

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

arXiv 2210.13382 ↗First posted 24 Oct 2022cs.LG cs.AI cs.CLcited by 417

Not on the map. Why?

Cited by 56 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

Key facts. Cited by 417 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

24 of 35 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2020Concept Bottleneck Modelscited by 1,022
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  12. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
12 more
  1. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  2. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  3. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  4. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  5. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  6. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  7. 2020Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoningcited by 65
  8. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  9. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  10. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  11. 2016Simpler Context-Dependent Logical Forms via Model Projectionscited by 33
  12. 2022Concept Gradient: Concept-based Interpretation Without Linear Assumptioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 68; 2024: 122; 2025: 100; 2026: 126.

417 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  3. 2023Steering Language Models With Activation Engineeringcited by 824
  4. 2023Large Language Modelscited by 774
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  9. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  12. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
38 more
  1. 2023Language Models Represent Space and Timecited by 306
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  3. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  4. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  5. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  6. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  7. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  8. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  9. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  10. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  11. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
  12. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  13. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  14. 2023The Quantization Model of Neural Scalingcited by 137
  15. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  16. 2024KAN 2.0: Kolmogorov-Arnold Networks Meet Sciencecited by 116
  17. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  18. 2024Evaluating the World Model Implicit in a Generative Modelcited by 107
  19. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  20. 2023Explaining grokking through circuit efficiencycited by 105
  21. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  22. 2023How do Language Models Bind Entities in Context?cited by 102
  23. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  24. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  25. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  26. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  27. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  28. 2024Robust agents learn causal world modelscited by 81
  29. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  30. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  31. 2023Eight Things to Know about Large Language Modelscited by 73
  32. 2024Transformers represent belief state geometry in their residual streamcited by 73
  33. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  34. 2023Could a Large Language Model be Conscious?cited by 70
  35. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 67
  36. 2023Learning to Model the World with Languagecited by 66
  37. 2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
  38. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 60

50 most cited of 417 All 417 (JSON)