mlsanityData to 7 Oct 2026

Acquisition of Chess Knowledge in AlphaZero

arXiv 2111.09259 ↗First posted 17 Nov 2021cs.AI stat.MLcited by 120

Not on the map. Why?

Abstract (arXiv)

What is learned by sophisticated neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 58 of its 100 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

58 of 100 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  8. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  9. 2018Sanity Checks for Saliency Mapscited by 1,149
  10. 2020Concept Bottleneck Modelscited by 1,022
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
38 more
  1. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  2. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  3. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  4. 2017The (Un)reliability of saliency methodscited by 331
  5. 2018On the Robustness of Interpretability Methodscited by 310
  6. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  7. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  8. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  9. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  10. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  11. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  12. 2020Debugging Tests for Model Explanationscited by 146
  13. 2017Visualizing and Understanding Atari Agentscited by 141
  14. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  15. 2021Promises and Pitfalls of Black-Box Concept Learning Modelscited by 128
  16. 2020Restricting the Flow: Information Bottlenecks for Attributioncited by 110
  17. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  18. 2020Explainability in Deep Reinforcement Learningcited by 96
  19. 2020Aligning Superhuman AI with Human Behavior: Chess as a Model Systemcited by 78
  20. 2018Relational Deep Reinforcement Learningcited by 72
  21. 2016Graying the black box: Understanding DQNscited by 68
  22. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 57
  23. 2020Towards falsifiable interpretability researchcited by 53
  24. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  25. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  26. 2016Towards Deep Symbolic Reinforcement Learningcited by 42
  27. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  28. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  29. 2020Debiasing Concept-based Explanations with Causal Analysiscited by 32
  30. 2019Regression Concept Vectors for Bidirectional Explanations in Histopathologycited by 31
  31. 2019Efficient Saliency Maps for Explainable AIcited by 30
  32. 2019Global and Local Interpretability for Cardiac MRI Classificationcited by 29
  33. 2019Gated Linear Networkscited by 26
  34. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  35. 2019Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based roboticscited by 21
  36. 2020Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chesscited by 15
  37. 2020Concept-based model explanations for Electronic Health Recordscited by 14
  38. 2019EDUCE: Explaining model Decisions through Unsupervised Concepts Extractioncited by 13

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 19; 2023: 33; 2024: 31; 2025: 24; 2026: 12.

120 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2022Post-hoc Concept Bottleneck Modelscited by 289
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  8. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  9. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  10. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  11. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  12. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
38 more
  1. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  2. 2024Robust agents learn causal world modelscited by 81
  3. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  4. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
  5. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  6. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  7. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  8. 2022On scientific understanding with artificial intelligencecited by 68
  9. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 60
  10. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Modelscited by 51
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  12. 2023Multi-dimensional concept discovery (MCD): A unifying framework with completeness guaranteescited by 48
  13. 2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 47
  14. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 41
  15. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  16. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 33
  17. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  18. 2023Disentangling Neuron Representations with Concept Vectorscited by 28
  19. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  20. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  21. 2022Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergencecited by 24
  22. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  23. 2024Maia-2: A Unified Model for Human-AI Alignment in Chesscited by 24
  24. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  25. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22
  26. 2024Position: Explain to Question not to Justifycited by 20
  27. 2024Opening the AI black box: program synthesis via mechanistic interpretabilitycited by 19
  28. 2024Planning in a recurrent neural network that plays Sokobancited by 17
  29. 2025Complete Chess Games Enable LLM Become A Chess Mastercited by 16
  30. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  31. 2025Hidden Breakthroughs in Language Model Trainingcited by 15
  32. 2024Mastering Chess with a Transformer Modelcited by 14
  33. 2024Transformers Use Causal World Models in Maze-Solving Taskscited by 13
  34. 2022Improving Chess Commentaries by Combining Language Models with Symbolic Reasoning Enginescited by 12
  35. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 12
  36. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  37. 2024Designing Skill-Compatible AI: Methodologies and Frameworks in Chesscited by 10
  38. 2023Q-SENN: Quantized Self-Explaining Neural Networkscited by 9

50 most cited of 120 All 120 (JSON)