# Deep Neuroevolution of Recurrent and Discrete World Models

arXiv: https://arxiv.org/abs/1906.08857 · HTML: https://mlsanity.com/paper/1906.08857/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 2 of its 41 references that are in the map.
- First posted on arXiv 28 Apr 2019.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 2 of 2)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)

## Built on by (most cited first, 23 of 23)

- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [HyperNCA: Growing Developmental Networks with Neural Cellular Automata](https://mlsanity.com/paper/2204.11674/) (2022)
- [Learning to Predict Without Looking Ahead: World Models Without Forward Prediction](https://mlsanity.com/paper/1910.13038/) (2019)
- [Proximal Policy Optimization with Continuous Bounded Action Space via the Beta Distribution](https://mlsanity.com/paper/2111.02202/) (2021)
- [Benchmarking Quality-Diversity Algorithms on Neuroevolution for Reinforcement Learning](https://mlsanity.com/paper/2211.02193/) (2022)
- [Evolutionary Strategies lead to Catastrophic Forgetting in LLMs](https://mlsanity.com/paper/2601.20861/) (2026)
- [Severe Damage Recovery in Evolving Soft Robots through Differentiable Programming](https://mlsanity.com/paper/2206.06674/) (2022)
- [Convolutional Reservoir Computing for World Models](https://mlsanity.com/paper/1907.08040/) (2019)
- [Deep Active Inference for Pixel-Based Discrete Control: Evaluation on the Car Racing Problem](https://mlsanity.com/paper/2109.04155/) (2021)
- [Direct evaluation of progression or regression of disease burden in brain metastatic disease with Deep Neuroevolution](https://mlsanity.com/paper/2203.12853/) (2022)
- [Hybrid Self-Attention NEAT: A novel evolutionary approach to improve the NEAT algorithm](https://mlsanity.com/paper/2112.03670/) (2021)
- [Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problems](https://mlsanity.com/paper/2201.12518/) (2022)
- [Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning](https://mlsanity.com/paper/2408.09493/) (2024)
- [Performance-Driven Controller Tuning via Derivative-Free Reinforcement Learning](https://mlsanity.com/paper/2209.04854/) (2022)
- [Evolving Artificial Neural Networks To Imitate Human Behaviour In Shinobi III : Return of the Ninja Master](https://mlsanity.com/paper/2304.01096/) (2023)
- [Neuroevolution of Recurrent Architectures on Control Tasks](https://mlsanity.com/paper/2304.12431/) (2023)
- [Generative Adversarial Neuroevolution for Control Behaviour Imitation](https://mlsanity.com/paper/2304.12432/) (2023)
- [Creating Hierarchical Dispositions of Needs in an Agent](https://mlsanity.com/paper/2412.00044/) (2024)
- [Low Rank Factorizations are Indirect Encodings for Deep Neuroevolution](https://mlsanity.com/paper/2504.03037/) (2025)
- [State Estimation Using Particle Filtering in Adaptive Machine Learning Methods: Integrating Q-Learning and NEAT Algorithms with Noisy Radar Measurements](https://mlsanity.com/paper/2504.07393/) (2025)

## Abstract (arXiv)

Neural architectures inspired by our own human cognitive system, such as the recently introduced world models, have been shown to outperform traditional deep reinforcement learning (RL) methods in a variety of different domains. Instead of the relatively simple architectures employed in most RL experiments, world models rely on multiple different neural components that are responsible for visual information processing, memory, and decision-making. However, so far the components of these models have to be trained separately and through a variety of specialized training methods. This paper demonstrates the surprising finding that models with the same precise parts can be instead efficiently trained end-to-end through a genetic algorithm (GA), reaching a comparable performance to the original world model by solving a challenging car racing task. An analysis of the evolved visual and memory system indicates that they include a similar effective representation to the system trained through gradient descent. Additionally, in contrast to gradient descent methods that struggle with discrete variables, GAs also work directly with such representations, opening up opportunities for classical planning in latent space. This paper adds additional evidence on the effectiveness of deep neuroevolution for tasks that require the intricate orchestration of multiple components in complex heterogeneous architectures.