# Smaller World Models for Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.05767 · HTML: https://mlsanity.com/paper/2010.05767/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 4 of its 13 references that are in the map.
- First posted on arXiv 12 Oct 2020.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 4 of 4)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)

## Built on by (most cited first, 4 of 4)

- [Deep Reinforcement Learning with Vector Quantized Encoding](https://mlsanity.com/paper/2211.06733/) (2022)
- [Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research](https://mlsanity.com/paper/2107.03015/) (2021)
- [Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering](https://mlsanity.com/paper/2409.17411/) (2024)
- [The Interpretability of Codebooks in Model-Based Reinforcement Learning is Limited](https://mlsanity.com/paper/2407.19532/) (2024)

## Abstract (arXiv)

Sample efficiency remains a fundamental issue of reinforcement learning. Model-based algorithms try to make better use of data by simulating the environment with a model. We propose a new neural network architecture for world models based on a vector quantized-variational autoencoder (VQ-VAE) to encode observations and a convolutional LSTM to predict the next embedding indices. A model-free PPO agent is trained purely on simulated experience from the world model. We adopt the setup introduced by Kaiser et al. (2020), which only allows 100K interactions with the real environment. We apply our method on 36 Atari environments and show that we reach comparable performance to their SimPLe algorithm, while our model is significantly smaller.