mlsanityData to 14 Oct 2026

Explore the Reasoning Capability of LLMs in the Chess Testbed

arXiv 2411.06655 ↗First posted 11 Nov 2024cs.CL cs.AIcited by 19

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Reasoning is a central capability of human intelligence. In recent years, with the advent of large-scale datasets, pretrained large language models have emerged with new capabilities, including reasoning. However, these models still struggle with long-term, complex reasoning tasks, such as playing chess. Based on the observation that expert chess players employ a dual approach combining long-term strategic play with short-term tactical play along with language explanation, we propose improving the reasoning capability of large language models in chess by integrating annotated strategy and tactic. Specifically, we collect a dataset named MATE, which consists of 1 million chess positions with candidate moves annotated by chess experts for strategy and tactics. We finetune the LLaMA-3-8B model and compare it against state-of-the-art commercial language models in the task of selecting better chess moves. Our experiments show that our models perform better than GPT, Claude, and Gemini models. We find that language explanations can enhance the reasoning capability of large language models.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 20 of its 29 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

20 of 29 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  8. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  9. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  10. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  11. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  12. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 49
8 more
  1. 2020The Chess Transformer: Mastering Play using Generative Language Modelscited by 37
  2. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  3. 2024Can Large Language Models Play Games? A Case Study of A Self-Play Approachcited by 13
  4. 2024Graph Neural Network Enhanced Retrieval for Question Answering of LLMscited by 10
  5. 2019SentiMATE: Learning to play Chess through Natural Language Processingcited by 9
  6. 2022Learning Chess With Language Models and Transformerscited by 6
  7. 2023Learning to Play Chess from Textbooks (LEAP): a Corpus for Evaluating Chess Moves based on Sentiment Analysiscited by 2
  8. 2024Learning the Latent Rules of a Game from Data: A Chess Storycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 8.

19 papers.

Sort:

By area:

  1. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  2. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  3. 2025TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compressioncited by 16
  4. 2025Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chesscited by 9
  5. 2026AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systemscited by 9
  6. 2025ChessQA: Evaluating Large Language Models for Chess Understandingcited by 8
  7. 2025R^3-VQA: "Read the Room" by Video Social Reasoningcited by 6
  8. 2025Xiangqi-R1: Enhancing Spatial Strategic Reasoning in LLMs for Chinese Chess via Reinforcement Learningcited by 5
  9. 2025ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Modelscited by 3
  10. 2025Beyond checkmate: exploring the creative chokepoints in AI textcited by 2
  11. 2026Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Gocited by 2
  12. 2025SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challengescited by 1
7 more
  1. 2026Disentangling generalization and memorization in large language models using chesscited by 1
  2. 2026Mixture of Masters: Sparse Chess Language Models with Player Routingcited by 1
  3. 2026GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectivescited by 1
  4. 2025Merge and Conquer: Evolutionarily Optimizing AI for 2048cited by 0
  5. 2026Se-DPO: Self-Evolving Token Credit for Direct Preference Optimizationcited by 0
  6. 2026Language Models that Play Chess and Explain Their Movescited by 0
  7. 2026Representation-Aligned Auxiliary Supervision for Language Model Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
  2. 2023Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skills
  3. 2026What Counts as Strategic Reasoning? A Systematic Mapping of Chess Research on Humans, Engines, and Language Models
  4. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  5. 2024Mastering Board Games by External and Internal Planning with Language Models
  6. 2026How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess
  7. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  8. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents