# Action Guidance with MCTS for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.11703 · HTML: https://mlsanity.com/paper/1907.11703/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2019.

## Built on by (most cited first, 8 of 8)

- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [Continuous Monte Carlo Graph Search](https://mlsanity.com/paper/2210.01426/) (2022)
- [MCTSteg: A Monte Carlo Tree Search-based Reinforcement Learning Framework for Universal Non-additive Steganography](https://mlsanity.com/paper/2103.13689/) (2021)
- [JiangJun: Mastering Xiangqi by Tackling Non-Transitivity in Two-Player Zero-Sum Games](https://mlsanity.com/paper/2308.04719/) (2023)
- [Online Robust Reinforcement Learning Through Monte-Carlo Planning](https://mlsanity.com/paper/2609.18599/) (2026)
- [LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search](https://mlsanity.com/paper/2603.21530/) (2026)
- [Exploring Adaptive MCTS with TD Learning in miniXCOM](https://mlsanity.com/paper/2210.05014/) (2022)
- [Enhancing Reinforcement Learning Through Guided Search](https://mlsanity.com/paper/2408.10113/) (2024)

## Abstract (arXiv)

Deep reinforcement learning has achieved great successes in recent years, however, one main challenge is the sample inefficiency. In this paper, we focus on how to use action guidance by means of a non-expert demonstrator to improve sample efficiency in a domain with sparse, delayed, and possibly deceptive rewards: the recently-proposed multi-agent benchmark of Pommerman. We propose a new framework where even a non-expert simulated demonstrator, e.g., planning algorithms such as Monte Carlo tree search with a small number rollouts, can be integrated within asynchronous distributed deep reinforcement learning methods. Compared to a vanilla deep RL algorithm, our proposed methods both learn faster and converge to better policies on a two-player mini version of the Pommerman game.