mlsanityData to 7 Oct 2026

Chess as a Testbed for Language Model State Tracking

arXiv 2102.13249 ↗First posted 26 Feb 2021cs.CL cs.AIcited by 69

Not on the map. Why?

Abstract (arXiv)

Transformer language models have made tremendous strides in natural language understanding tasks. However, the complexity of natural language makes it challenging to ascertain how accurately these models are tracking the world state underlying the text. Motivated by this issue, we consider the task of language modeling for the game of chess. Unlike natural language, chess notations describe a simple, constrained, and deterministic domain. Moreover, we observe that the appropriate choice of chess notation allows for directly probing the world state, without requiring any additional probing-related machinery. We find that: (a) With enough training data, transformer language models can learn to track pieces and predict legal moves with high accuracy when trained solely on move sequences. (b) For small training sets providing access to board state information during training can yield significant improvements. (c) The success of transformer language models is dependent on access to the entire game history i.e. "full attention". Approximating this full attention results in a significant performance drop. We propose this testbed as a benchmark for future work on the development and analysis of transformer language models.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

27 of 40 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2017Mixed Precision Trainingcited by 1,135
15 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,039
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  3. 2019Designing and Interpreting Probes with Control Taskscited by 608
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  6. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  7. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  8. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  9. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  11. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  12. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  13. 2020The Chess Transformer: Mastering Play using Generative Language Modelscited by 36
  14. 2017DeepChess: End-to-End Deep Neural Network for Automatic Learning in Chesscited by 12
  15. 2017Understanding Early Word Learning in Situated Artificial Agentscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 22; 2025: 17; 2026: 18.

69 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2023Language Models Represent Space and Timecited by 306
  3. 2024The Illusion of State in State-Space Modelscited by 181
  4. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  5. 2024Evaluating the World Model Implicit in a Generative Modelcited by 107
  6. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  7. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  8. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  9. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
  10. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 46
  11. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 45
  12. 2023Entity Tracking in Language Modelscited by 44
38 more
  1. 2024A Language Model's Guide Through Latent Spacecited by 43
  2. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  3. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 33
  4. 2023Learning to Reason and Memorize with Self-Notescited by 30
  5. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 28
  6. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22
  7. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  8. 2024Opening the AI black box: program synthesis via mechanistic interpretabilitycited by 19
  9. 2024Transcendence: Generative Models Can Outperform The Experts That Train Themcited by 18
  10. 2025Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Trackingcited by 17
  11. 2022Improving Chess Commentaries by Combining Language Models with Symbolic Reasoning Enginescited by 12
  12. 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chesscited by 12
  13. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  14. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 10
  15. 2024GPT for Games: An Updated Scoping Review (2020-2024)cited by 10
  16. 2023Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skillscited by 9
  17. 2024Human-aligned Chess with a Bit of Searchcited by 9
  18. 2023Representation Matters for Mastering Chess: Improved Feature Representation in AlphaZero Outperforms Switching to Transformerscited by 8
  19. 2024Chain and Causal Attention for Efficient Entity Trackingcited by 8
  20. 2025What Does it Mean for a Neural Network to Learn a "World Model"?cited by 7
  21. 2024A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environmentcited by 6
  22. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  23. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  24. 2026Learning State-Tracking from Code Using Linear RNNscited by 5
  25. 2026Do Language Models Track Entities Across State Changes?cited by 5
  26. 2023Can You Follow Me? Testing Situational Understanding in ChatGPTcited by 4
  27. 2025Linear Spatial World Models Emerge in Large Language Modelscited by 4
  28. 2025Tracking World States with Language Models: State-Based Evaluation Using Chesscited by 4
  29. 2025The Limits of Predicting Agents from Behaviourcited by 3
  30. 2026TopoBench: Benchmarking LLMs on Hard Topological Reasoningcited by 3
  31. 2024Reasoning, Memorization, and Fine-Tuning Language Models for Non-Cooperative Gamescited by 2
  32. 2025From Next Token Prediction to (STRIPS) World Modelscited by 2
  33. 2025The Impossibility of Inverse Permutation Learning in Transformer Modelscited by 2
  34. 2025Out-of-distribution Tests Reveal Compositionality in Chess Transformerscited by 2
  35. 2026Can Vision-Language Models Solve the Shell Game?cited by 2
  36. 2024Aspect-based Sentiment Evaluation of Chess Moves (ASSESS): an NLP-based Method for Evaluating Chess Strategies from Textbookscited by 1
  37. 2024A Text-to-Game Engine for UGC-Based Role-Playing Gamescited by 1
  38. 2024Exploring the Learning Capabilities of Language Models using LEVERWORLDScited by 1

50 most cited of 69 All 69 (JSON)