# Continual Learning Using World Models for Pseudo-Rehearsal

arXiv: https://arxiv.org/abs/1903.02647 · HTML: https://mlsanity.com/paper/1903.02647/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 3 of its 25 references that are in the map.
- First posted on arXiv 6 Mar 2019.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 3 of 3)

- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models](https://mlsanity.com/paper/1511.09249/) (2015)

## Built on by (most cited first, 4 of 4)

- [Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2](https://mlsanity.com/paper/2208.05056/) (2022)
- [Knowledge Retention for Continual Model-Based Reinforcement Learning](https://mlsanity.com/paper/2503.04256/) (2025)
- [Integrating Curricula with Replays: Its Effects on Continual Learning](https://mlsanity.com/paper/2307.05747/) (2023)
- [Concept-modulated model-based offline reinforcement learning for rapid generalization](https://mlsanity.com/paper/2209.03207/) (2022)

## Abstract (arXiv)

The utility of learning a dynamics/world model of the environment in reinforcement learning has been shown in a many ways. When using neural networks, however, these models suffer catastrophic forgetting when learned in a lifelong or continual fashion. Current solutions to the continual learning problem require experience to be segmented and labeled as discrete tasks, however, in continuous experience it is generally unclear what a sufficient segmentation of tasks would be. Here we propose a method to continually learn these internal world models through the interleaving of internally generated episodes of past experiences (i.e., pseudo-rehearsal). We show this method can sequentially learn unsupervised temporal prediction, without task labels, in a disparate set of Atari games. Empirically, this interleaving of the internally generated rollouts with the external environment's observations leads to a consistent reduction in temporal prediction loss compared to non-interleaved learning and is preserved over repeated random exposures to various tasks. Similarly, using a network distillation approach, we show that modern policy gradient based reinforcement learning algorithms can use this internal model to continually learn to optimize reward based on the world model's representation of the environment.