# AlphaZero-Inspired Game Learning: Faster Training by Using MCTS Only at Test Time

arXiv: https://arxiv.org/abs/2204.13307 · HTML: https://mlsanity.com/paper/2204.13307/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 10 of 10)

- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Ludii -- The Ludemic General Game System](https://mlsanity.com/paper/1905.05013/) (2019)
- [Polygames: Improved Zero Learning](https://mlsanity.com/paper/2001.09832/) (2020)
- [Deep Learning for General Game Playing with Ludii and Polygames](https://mlsanity.com/paper/2101.09562/) (2021)
- [Learning to Play Othello with Deep Neural Networks](https://mlsanity.com/paper/1711.06583/) (2017)
- [Mastering 2048 with Delayed Temporal Coherence Learning, Multi-Stage Weight Promotion, Redundant Encoding and Carousel Shaping](https://mlsanity.com/paper/1604.05085/) (2016)
- [Warm-Start AlphaZero Self-Play Search Enhancements](https://mlsanity.com/paper/2004.12357/) (2020)
- [Hyper-Parameter Sweep on AlphaZero General](https://mlsanity.com/paper/1903.08129/) (2019)
- [General Board Game Playing for Education and Research in Generic AI Game Learning](https://mlsanity.com/paper/1907.06508/) (2019)
- [Final Adaptation Reinforcement Learning for N-Player Games](https://mlsanity.com/paper/2111.14375/) (2021)

## Built on by (most cited first, 3 of 3)

- [An Evolutionary Framework for Connect-4 as Test-Bed for Comparison of Advanced Minimax, Q-Learning and MCTS](https://mlsanity.com/paper/2405.16595/) (2024)
- [Towards Learning Rubik's Cube with N-tuple-based Reinforcement Learning](https://mlsanity.com/paper/2301.12167/) (2023)
- [AlphaViT: A flexible game-playing AI for multiple games and variable board sizes](https://mlsanity.com/paper/2408.13871/) (2024)

## Abstract (arXiv)

Recently, the seminal algorithms AlphaGo and AlphaZero have started a new era in game learning and deep reinforcement learning. While the achievements of AlphaGo and AlphaZero - playing Go and other complex games at super human level - are truly impressive, these architectures have the drawback that they require high computational resources. Many researchers are looking for methods that are similar to AlphaZero, but have lower computational demands and are thus more easily reproducible. In this paper, we pick an important element of AlphaZero - the Monte Carlo Tree Search (MCTS) planning stage - and combine it with temporal difference (TD) learning agents. We wrap MCTS for the first time around TD n-tuple networks and we use this wrapping only at test time to create versatile agents that keep at the same time the computational demands low. We apply this new architecture to several complex games (Othello, ConnectFour, Rubik's Cube) and show the advantages achieved with this AlphaZero-inspired MCTS wrapper. In particular, we present results that this agent is the first one trained on standard hardware (no GPU or TPU) to beat the very strong Othello program Edax up to and including level 7 (where most other learning-from-scratch algorithms could only defeat Edax up to level 2).