# Brick Tic-Tac-Toe: Exploring the Generalizability of AlphaZero to Novel Test Environments

arXiv: https://arxiv.org/abs/2207.05991 · HTML: https://mlsanity.com/paper/2207.05991/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Monte-Carlo Tree Search as Regularized Policy Optimization](https://mlsanity.com/paper/2007.12509/) (2020)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero](https://mlsanity.com/paper/1902.04522/) (2019)
- [A0C: Alpha Zero in Continuous Action Space](https://mlsanity.com/paper/1805.09613/) (2018)
- [Minimalistic Attacks: How Little it Takes to Fool a Deep Reinforcement Learning Policy](https://mlsanity.com/paper/1911.03849/) (2019)
- [Convex Regularization in Monte-Carlo Tree Search](https://mlsanity.com/paper/2007.00391/) (2020)
- [Transfer of Fully Convolutional Policy-Value Networks Between Games and Game Variants](https://mlsanity.com/paper/2102.12375/) (2021)
- [Train on Small, Play the Large: Scaling Up Board Games with AlphaZero and GNN](https://mlsanity.com/paper/2107.08387/) (2021)
- [Hyper-Parameter Sweep on AlphaZero General](https://mlsanity.com/paper/1903.08129/) (2019)

## Built on by (most cited first, 2 of 2)

- [An Approach to Solving the Abstraction and Reasoning Corpus (ARC) Challenge](https://mlsanity.com/paper/2306.03553/) (2023)
- [Improving Robustness of AlphaZero Algorithms to Test-Time Environment Changes](https://mlsanity.com/paper/2509.04317/) (2025)

## Abstract (arXiv)

Traditional reinforcement learning (RL) environments typically are the same for both the training and testing phases. Hence, current RL methods are largely not generalizable to a test environment which is conceptually similar but different from what the method has been trained on, which we term the novel test environment. As an effort to push RL research towards algorithms which can generalize to novel test environments, we introduce the Brick Tic-Tac-Toe (BTTT) test bed, where the brick position in the test environment is different from that in the training environment. Using a round-robin tournament on the BTTT environment, we show that traditional RL state-search approaches such as Monte Carlo Tree Search (MCTS) and Minimax are more generalizable to novel test environments than AlphaZero is. This is surprising because AlphaZero has been shown to achieve superhuman performance in environments such as Go, Chess and Shogi, which may lead one to think that it performs well in novel test environments. Our results show that BTTT, though simple, is rich enough to explore the generalizability of AlphaZero. We find that merely increasing MCTS lookahead iterations was insufficient for AlphaZero to generalize to some novel test environments. Rather, increasing the variety of training environments helps to progressively improve generalizability across all possible starting brick configurations.