# Mastering Chess with a Transformer Model

arXiv: https://arxiv.org/abs/2409.12272 · HTML: https://mlsanity.com/paper/2409.12272/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Accelerating Self-Play Learning in Go](https://mlsanity.com/paper/1902.10565/) (2019)
- [Evidence of Learned Look-Ahead in a Chess-Playing Neural Network](https://mlsanity.com/paper/2406.00877/) (2024)
- [Representation Matters for Mastering Chess: Improved Feature Representation in AlphaZero Outperforms Switching to Transformers](https://mlsanity.com/paper/2304.14918/) (2023)

## Built on by (most cited first, 15 of 15)

- [LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess](https://mlsanity.com/paper/2512.01992/) (2025)
- [LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Needed](https://mlsanity.com/paper/2411.08794/) (2024)
- [Scalable Decision Making for Games of Imperfect Information](https://mlsanity.com/paper/2511.07312/) (2025)
- [Grounded Chess Reasoning in Language Models via Master Distillation](https://mlsanity.com/paper/2603.20510/) (2026)
- [Predicting Chess Puzzle Difficulty with Transformers](https://mlsanity.com/paper/2410.11078/) (2024)
- [Out-of-distribution Tests Reveal Compositionality in Chess Transformers](https://mlsanity.com/paper/2510.20783/) (2025)
- [Exploring Human-AI Conceptual Alignment through the Prism of Chess](https://mlsanity.com/paper/2510.26025/) (2025)
- [Tracing the Thought of a Grandmaster-level Chess-Playing Transformer](https://mlsanity.com/paper/2604.10158/) (2026)
- [TransZero: Parallel Tree Expansion in MuZero using Transformer Networks](https://mlsanity.com/paper/2509.11233/) (2025)
- [Mixture of Masters: Sparse Chess Language Models with Player Routing](https://mlsanity.com/paper/2602.04447/) (2026)
- [Matilda: Engine-Agnostic Search with Human Policy Guidance](https://mlsanity.com/paper/2606.25176/) (2026)
- [Exploring Collaboration between a language and a non-language agent](https://mlsanity.com/paper/2609.00474/) (2026)
- [Bridging Local and Global Knowledge via Transformer in Board Games](https://mlsanity.com/paper/2410.05347/) (2024)
- [UniMaia: Steering Chess Policies with Language for Human-like Play](https://mlsanity.com/paper/2605.27767/) (2026)
- [Otter: A Time-Aware, History-Conditioned Human Chess AI](https://mlsanity.com/paper/2608.05206/) (2026)

## Abstract (arXiv)

Transformer models have demonstrated impressive capabilities when trained at scale, excelling at difficult cognitive tasks requiring complex reasoning and rational decision-making. In this paper, we explore the application of transformers to chess, focusing on the critical role of the position representation within the attention mechanism. We show that transformers endowed with a sufficiently expressive position representation can match existing chess-playing models at a fraction of the computational cost. Our architecture, which we call the Chessformer, significantly outperforms AlphaZero in both playing strength and puzzle solving ability with 8x less computation and matches prior grandmaster-level transformer-based agents in those metrics with 30x less computation. Our models also display an understanding of chess dissimilar and orthogonal to that of top traditional engines, detecting high-level positional features like trapped pieces and fortresses that those engines struggle with. This work demonstrates that domain-specific enhancements can in large part replace the need for model scale, while also highlighting that deep learning can make strides even in areas dominated by search-based methods.