# Evolutionary Planning in Latent Space

arXiv: https://arxiv.org/abs/2011.11293 · HTML: https://mlsanity.com/paper/2011.11293/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 23 Nov 2020.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 9 of 9)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [Neural Game Engine: Accurate learning of generalizable forward models from pixels](https://mlsanity.com/paper/2003.10520/) (2020)
- [Rolling Horizon Evolutionary Algorithms for General Video Game Playing](https://mlsanity.com/paper/2003.12331/) (2020)
- [Bootstrapped model learning and error correction for planning with uncertainty in model-based RL](https://mlsanity.com/paper/2004.07155/) (2020)

## Built on by (most cited first, 3 of 3)

- [LaPlaSS: Latent Space Planning for Stochastic Systems](https://mlsanity.com/paper/2404.07063/) (2024)
- [Predictive Control Using Learned State Space Models via Rolling Horizon Evolution](https://mlsanity.com/paper/2106.13911/) (2021)
- [Evolutionary Machine Learning and Games](https://mlsanity.com/paper/2311.16172/) (2023)

## Abstract (arXiv)

Planning is a powerful approach to reinforcement learning with several desirable properties. However, it requires a model of the world, which is not readily available in many real-life problems. In this paper, we propose to learn a world model that enables Evolutionary Planning in Latent Space (EPLS). We use a Variational Auto Encoder (VAE) to learn a compressed latent representation of individual observations and extend a Mixture Density Recurrent Neural Network (MDRNN) to learn a stochastic, multi-modal forward model of the world that can be used for planning. We use the Random Mutation Hill Climbing (RMHC) to find a sequence of actions that maximize expected reward in this learned model of the world. We demonstrate how to build a model of the world by bootstrapping it with rollouts from a random policy and iteratively refining it with rollouts from an increasingly accurate planning policy using the learned world model. After a few iterations of this refinement, our planning agents are better than standard model-free reinforcement learning approaches demonstrating the viability of our approach.