# Is Policy Learning Overrated?: Width-Based Planning and Active Learning for Atari

arXiv: https://arxiv.org/abs/2109.15310 · HTML: https://mlsanity.com/paper/2109.15310/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Simple Bayesian Algorithms for Best Arm Identification](https://mlsanity.com/paper/1602.08448/) (2016)
- [Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary](https://mlsanity.com/paper/1705.00154/) (2017)
- [State of the Art Control of Atari Games Using Shallow Reinforcement Learning](https://mlsanity.com/paper/1512.01563/) (2015)
- [Understanding the Power and Limitations of Teaching with Imperfect Knowledge](https://mlsanity.com/paper/2003.09712/) (2020)
- [Classical Planning in Deep Latent Space](https://mlsanity.com/paper/2107.00110/) (2021)
- [Monte-Carlo Tree Search by Best Arm Identification](https://mlsanity.com/paper/1706.02986/) (2017)
- [Planning from Pixels in Atari with Learned Symbolic Representations](https://mlsanity.com/paper/2012.09126/) (2020)
- [On Deep Unsupervised Active Learning](https://mlsanity.com/paper/2007.13959/) (2020)
- [Deep Policies for Width-Based Planning in Pixel Domains](https://mlsanity.com/paper/1904.07091/) (2019)
- [Hierarchical Width-Based Planning and Learning](https://mlsanity.com/paper/2101.06177/) (2021)
- [Planning with Pixels in (Almost) Real Time](https://mlsanity.com/paper/1801.03354/) (2018)
- [Towards Stable Symbol Grounding with Zero-Suppressed State AutoEncoder](https://mlsanity.com/paper/1903.11277/) (2019)

## Built on by (most cited first, 1 of 1)

- [Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RL](https://mlsanity.com/paper/2410.17428/) (2024)

## Abstract (arXiv)

Width-based planning has shown promising results on Atari 2600 games using pixel input, while using substantially fewer environment interactions than reinforcement learning. Recent width-based approaches have computed feature vectors for each screen using a hand designed feature set or a variational autoencoder trained on game screens (VAE-IW), and prune screens that do not have novel features during the search. We propose Olive (Online-VAE-IW), which updates the VAE features online using active learning to maximize the utility of screens observed during planning. Experimental results in 55 Atari games demonstrate that it outperforms Rollout-IW by 42-to-11 and VAE-IW by 32-to-20. Moreover, Olive outperforms existing work based on policy-learning (π-IW, DQN) trained with 100x training budget by 30-to-22 and 31-to-17, and a state of the art data-efficient reinforcement learning (EfficientZero) trained with the same training budget and ran with 1.8x planning budget by 18-to-7 in Atari 100k benchmark, with no policy learning at all. The source code is available at github.com/ibm/atari-active-learning .