mlsanityData to 14 Oct 2026

Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero

arXiv 2310.16410 ↗First posted 25 Oct 2023cs.AI cs.HC cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Artificial Intelligence (AI) systems have made remarkable progress, attaining super-human performance across various domains. This presents us with an opportunity to further human knowledge and improve human expert performance by leveraging the hidden knowledge encoded within these highly performant AI systems. Yet, this knowledge is often hard to extract, and may be hard to understand or learn from. Here, we show that this is possible by proposing a new method that allows us to extract new chess concepts in AlphaZero, an AI system that mastered the game of chess via self-play without human supervision. Our analysis indicates that AlphaZero may encode knowledge that extends beyond the existing human knowledge, but knowledge that is ultimately not beyond human grasp, and can be successfully learned from. In a human study, we show that these concepts are learnable by top human experts, as four top chess grandmasters show improvements in solving the presented concept prototype positions. This marks an important first milestone in advancing the frontier of human knowledge by leveraging AI; a development that could bear profound implications and help us shape how we interact with AI systems across many AI applications.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 59 of its 127 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

59 of 127 references.

References · 59

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2020Concept Bottleneck Modelscited by 1,032
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  10. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
38 more
  1. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  2. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  3. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  4. 2021Promises and Pitfalls of Black-Box Concept Learning Modelscited by 131
  5. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  6. 2022Impossibility Theorems for Feature Attributioncited by 78
  7. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 75
  8. 2018Relational Deep Reinforcement Learningcited by 73
  9. 2021A Survey on Interpretable Reinforcement Learningcited by 73
  10. 2016Graying the black box: Understanding DQNscited by 68
  11. 2018A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizationscited by 60
  12. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  13. 2022Concept Activation Regions: A Generalized Framework For Concept-Based Explanationscited by 57
  14. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  15. 2016Towards Deep Symbolic Reinforcement Learningcited by 42
  16. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  17. 2021DISSECT: Disentangled Simultaneous Explanations via Concept Traversalscited by 37
  18. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 36
  19. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  20. 2020Debiasing Concept-based Explanations with Causal Analysiscited by 32
  21. 2022A Survey of Explainable Reinforcement Learningcited by 31
  22. 2019Efficient Saliency Maps for Explainable AIcited by 30
  23. 2019Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policycited by 29
  24. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  25. 2019Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based roboticscited by 21
  26. 2022Concept Gradient: Concept-based Interpretation Without Linear Assumptioncited by 20
  27. 2023State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understandingcited by 17
  28. 2019Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agentscited by 15
  29. 2020Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chesscited by 15
  30. 2023Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approachcited by 15
  31. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  32. 2019Counterfactual States for Atari Agents via Generative Deep Learningcited by 14
  33. 2022Are AlphaZero-like Agents Robust to Adversarial Perturbations?cited by 14
  34. 2019EDUCE: Explaining model Decisions through Unsupervised Concepts Extractioncited by 13
  35. 2020Distal Explanations for Model-free Explainable Reinforcement Learningcited by 12
  36. 2020Adversarial TCAV -- Robust and Effective Interpretation of Intermediate Layers in Neural Networkscited by 12
  37. 2019Towards Automatic Concept-based Explanationscited by 11
  38. 2022Explainability in reinforcement learning: perspective and positioncited by 11

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 13; 2026: 8.

42 papers.

Sort:

By area:

  1. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  2. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  3. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 34
  4. 2024On the Challenges and Opportunities in Generative AIcited by 31
  5. 2025Characterizing AI Agents for Alignment and Governancecited by 30
  6. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  7. 2024Maia-2: A Unified Model for Human-AI Alignment in Chesscited by 25
  8. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debatescited by 23
  9. 2024Planning in a recurrent neural network that plays Sokobancited by 18
  10. 2024Natural Language Reinforcement Learningcited by 17
  11. 2024Neural Concept Bindercited by 14
  12. 2025Towards Interpretable Deep Generative Models via Causal Representation Learningcited by 13
30 more
  1. 2024Interpretability Needs a New Paradigmcited by 11
  2. 2025Decomposing Elements of Problem Solving: What "Math" Does RL Teach?cited by 10
  3. 2024Rethinking Explainable Machine Learning as Applied Statisticscited by 8
  4. 2024A survey on Concept-based Approaches For Model Improvementcited by 8
  5. 2025Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selectioncited by 8
  6. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  7. 2024Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationshipscited by 7
  8. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  9. 2025Learning to Imitate with Less: Efficient Individual Behavior Modeling in Chesscited by 7
  10. 2023Physics simulation capabilities of LLMscited by 6
  11. 2024Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluationcited by 6
  12. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  13. 2025A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.icited by 6
  14. 2024Natural Language Reinforcement Learningcited by 5
  15. 2025When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaborationcited by 4
  16. 2025Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNNcited by 4
  17. 2026Sanity Checks for Agentic Data Sciencecited by 4
  18. 2024Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representationscited by 3
  19. 2024Contrastive Sparse Autoencoders for Interpreting Planning of Chess-Playing Agentscited by 2
  20. 2026Emerging Human-like Strategies for Semantic Memory Foraging in Large Language Modelscited by 2
  21. 2025Concept activation vectors: a unifying view and adversarial attackscited by 1
  22. 2025Visual Interestingness Decoded: How GPT-4o Mirrors Human Interestscited by 1
  23. 2025Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Modelscited by 1
  24. 2024Concept Learning in the Wild: Towards Algorithmic Understanding of Neural Networkscited by 0
  25. 2025Example-Based Concept Analysis Framework for Deep Weather Forecast Modelscited by 0
  26. 2026Concept Component Analysis: A Principled Approach for Concept Extraction in LLMscited by 0
  27. 2026α-TCAV: A Unified Framework for Testing with Concept Activation Vectorscited by 0
  28. 2026Human Decision-Making with AI Assistance under Correlated Featurescited by 0
  29. 2026EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failurescited by 0
  30. 2026Discovering High-Quality Chess Puzzles with Offline Reinforcement Learningcited by 0