# Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games

arXiv: https://arxiv.org/abs/1604.07095 · HTML: https://mlsanity.com/paper/1604.07095/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2016.

## Built on by (most cited first, 13 of 13)

- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [Planning Like Human: A Dual-process Framework for Dialogue Planning](https://mlsanity.com/paper/2406.05374/) (2024)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Generative Adversarial Self-Imitation Learning](https://mlsanity.com/paper/1812.00950/) (2018)
- [Redeeming Intrinsic Rewards via Constrained Optimization](https://mlsanity.com/paper/2211.07627/) (2022)
- [Provably Efficient Policy Optimization for Two-Player Zero-Sum Markov Games](https://mlsanity.com/paper/2102.08903/) (2021)
- [Behavior Alignment via Reward Function Optimization](https://mlsanity.com/paper/2310.19007/) (2023)
- [The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning](https://mlsanity.com/paper/2302.13493/) (2023)
- [Learning the Reward Function for a Misspecified Model](https://mlsanity.com/paper/1801.09624/) (2018)
- [Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.04819/) (2023)
- [Turning Mathematics Problems into Games: Reinforcement Learning and Gr"obner bases together solve Integer Feasibility Problems](https://mlsanity.com/paper/2208.12191/) (2022)
- [Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks](https://mlsanity.com/paper/2107.06405/) (2021)
- [Policy Gradient Algorithms with Monte Carlo Tree Learning for Non-Markov Decision Processes](https://mlsanity.com/paper/2206.01011/) (2022)

## Abstract (arXiv)

Monte Carlo Tree Search (MCTS) methods have proven powerful in planning for sequential decision-making problems such as Go and video games, but their performance can be poor when the planning depth and sampling trajectories are limited or when the rewards are sparse. We present an adaptation of PGRD (policy-gradient for reward-design) for learning a reward-bonus function to improve UCT (a MCTS algorithm). Unlike previous applications of PGRD in which the space of reward-bonus functions was limited to linear functions of hand-coded state-action-features, we use PGRD with a multi-layer convolutional neural network to automatically learn features from raw perception as well as to adapt the non-linear reward-bonus function parameters. We also adopt a variance-reducing gradient method to improve PGRD's performance. The new method improves UCT's performance on multiple ATARI games compared to UCT without the reward bonus. Combining PGRD and Deep Learning in this way should make adapting rewards for MCTS algorithms far more widely and practically applicable than before.