mlsanityData to 14 Oct 2026

ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models

arXiv 2509.24239 ↗First posted 29 Sep 2025cs.LG cs.AIcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Recent large language models (LLMs) have shown strong reasoning capabilities. However, a critical question remains: do these models possess genuine strategic reasoning, or do they primarily excel at pattern recognition? To address this, we present ChessArena, a chess-based testbed for evaluating LLMs. Chess demands strategic reasoning, precise rule adherence, and the ability to track complex game states. ChessArena is a competitive framework where LLMs play against each other under four play modes. We evaluate 13 LLMs across over 800 games, testing basic understanding, move selection, and puzzle solving. Results reveal significant shortcomings: no model beats Maia-1100 (human amateur level), and some lose to random play. We also present a strong baseline: our fine-tuned Qwen3-8B substantially improves performance, approaching much larger state-of-the-art reasoning models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 31 of its 44 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

31 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Xiangqi-R1: Enhancing Spatial Strategic Reasoning in LLMs for Chinese Chess via Reinforcement Learning16 Jul
  2. 2025Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess1 Jul
  3. 2024Explore the Reasoning Capability of LLMs in the Chess Testbed11 Nov
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  5. 2023Strategic Reasoning with Language Models30 May

Other references · 26

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
14 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  4. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  5. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  6. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  7. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  8. 2024Continual Learning with Pre-Trained Models: A Surveycited by 154
  9. 2025BIG-Bench Extra Hardcited by 125
  10. 2020Aligning Superhuman AI with Human Behavior: Chess as a Model Systemcited by 79
  11. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 75
  12. 2024FullStack Bench: Evaluating LLMs as Full Stack Coderscited by 58
  13. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 49
  14. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planningcited by 2
  2. 2026Learning Stateful Predictive Knowledge From Experiencecited by 2
  3. 2026Benchmarking Prompt Optimization of Large Language Models With Chesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
  2. 2023Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skills
  3. 2026What Counts as Strategic Reasoning? A Systematic Mapping of Chess Research on Humans, Engines, and Language Models
  4. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  5. 2024Mastering Board Games by External and Internal Planning with Language Models
  6. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  7. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  8. 2025Tracking World States with Language Models: State-Based Evaluation Using Chess