# Know your Enemy: Investigating Monte-Carlo Tree Search with Opponent Models in Pommerman

arXiv: https://arxiv.org/abs/2305.13206 · HTML: https://mlsanity.com/paper/2305.13206/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 15 of 15)

- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [Accelerating Self-Play Learning in Go](https://mlsanity.com/paper/1902.10565/) (2019)
- [Vector Quantized Models for Planning](https://mlsanity.com/paper/2106.04615/) (2021)
- [Pommerman: A Multi-Agent Playground](https://mlsanity.com/paper/1809.07124/) (2018)
- [Backplay: "Man muss immer umkehren"](https://mlsanity.com/paper/1807.06919/) (2018)
- [Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition](https://mlsanity.com/paper/1905.01360/) (2019)
- [Multiplayer AlphaZero](https://mlsanity.com/paper/1910.13012/) (2019)
- [Learning to play the Chess Variant Crazyhouse above World Champion Level with Deep Neural Networks and Human Data](https://mlsanity.com/paper/1908.06660/) (2019)
- [Real-time tree search with pessimistic scenarios](https://mlsanity.com/paper/1902.10870/) (2019)
- [On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman](https://mlsanity.com/paper/1907.11788/) (2019)
- [Continual Match Based Training in Pommerman: Technical Report](https://mlsanity.com/paper/1812.07297/) (2018)
- [Safer Deep RL with Shallow MCTS: A Case Study in Pommerman](https://mlsanity.com/paper/1904.05759/) (2019)

## Abstract (arXiv)

In combination with Reinforcement Learning, Monte-Carlo Tree Search has shown to outperform human grandmasters in games such as Chess, Shogi and Go with little to no prior domain knowledge. However, most classical use cases only feature up to two players. Scaling the search to an arbitrary number of players presents a computational challenge, especially if decisions have to be planned over a longer time horizon. In this work, we investigate techniques that transform general-sum multiplayer games into single-player and two-player games that consider other agents to act according to given opponent models. For our evaluation, we focus on the challenging Pommerman environment which involves partial observability, a long time horizon and sparse rewards. In combination with our search methods, we investigate the phenomena of opponent modeling using heuristics and self-play. Overall, we demonstrate the effectiveness of our multiplayer search variants both in a supervised learning and reinforcement learning setting.