# Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting

arXiv: https://arxiv.org/abs/1812.02464 · HTML: https://mlsanity.com/paper/1812.02464/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2018.

## Built on by (most cited first, 30 of 46)

- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [Advanced Skills through Multiple Adversarial Motion Priors in Reinforcement Learning](https://mlsanity.com/paper/2203.14912/) (2022)
- [Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations](https://mlsanity.com/paper/2108.05851/) (2021)
- [Building a Subspace of Policies for Scalable Continual Learning](https://mlsanity.com/paper/2211.10445/) (2022)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Learn and Don't Forget: Adding a New Language to ASR Foundation Models](https://mlsanity.com/paper/2407.06800/) (2024)
- [Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation](https://mlsanity.com/paper/2109.00525/) (2021)
- [CoMPS: Continual Meta Policy Search](https://mlsanity.com/paper/2112.04467/) (2021)
- [Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization](https://mlsanity.com/paper/2209.00347/) (2022)
- [Unveiling the Anomalies in an Ever-Changing World: A Benchmark for Pixel-Level Anomaly Detection in Continual Learning](https://mlsanity.com/paper/2403.15463/) (2024)
- [Continual Learning on Graphs: A Survey](https://mlsanity.com/paper/2402.06330/) (2024)
- [Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2](https://mlsanity.com/paper/2208.05056/) (2022)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.14990/) (2025)
- [Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation](https://mlsanity.com/paper/2205.10868/) (2022)
- [Continual Learning Approaches for Anomaly Detection](https://mlsanity.com/paper/2212.11192/) (2022)
- [Active Fine-Tuning of Multi-Task Policies](https://mlsanity.com/paper/2410.05026/) (2024)
- [Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks](https://mlsanity.com/paper/2401.15275/) (2024)
- [Replay-enhanced Continual Reinforcement Learning](https://mlsanity.com/paper/2311.11557/) (2023)
- [Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay](https://mlsanity.com/paper/2411.10809/) (2024)
- [Online Reinforcement Learning in Non-Stationary Context-Driven Environments](https://mlsanity.com/paper/2302.02182/) (2023)
- [Certified Continual Learning for Neural Network Regression](https://mlsanity.com/paper/2407.06697/) (2024)
- [Overcoming Catastrophic Forgetting in Tabular Data Classification: A Pseudorehearsal-based approach](https://mlsanity.com/paper/2407.09039/) (2024)
- [ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2412.16848/) (2024)
- [Avalanche RL: a Continual Reinforcement Learning Library](https://mlsanity.com/paper/2202.13657/) (2022)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [CLVOS23: A Long Video Object Segmentation Dataset for Continual Learning](https://mlsanity.com/paper/2304.04259/) (2023)
- [Demystifying Reinforcement Learning in Time-Varying Systems](https://mlsanity.com/paper/2201.05560/) (2022)
- [Elastic Multi-Gradient Descent for Parallel Continual Learning](https://mlsanity.com/paper/2401.01054/) (2024)
- [Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal](https://mlsanity.com/paper/2409.02512/) (2024)

## Abstract (arXiv)

Neural networks can achieve excellent results in a wide variety of applications. However, when they attempt to sequentially learn, they tend to learn the new task while catastrophically forgetting previous ones. We propose a model that overcomes catastrophic forgetting in sequential reinforcement learning by combining ideas from continual learning in both the image classification domain and the reinforcement learning domain. This model features a dual memory system which separates continual learning from reinforcement learning and a pseudo-rehearsal system that "recalls" items representative of previous tasks via a deep generative network. Our model sequentially learns Atari 2600 games without demonstrating catastrophic forgetting and continues to perform above human level on all three games. This result is achieved without: demanding additional storage requirements as the number of tasks increases, storing raw data or revisiting past tasks. In comparison, previous state-of-the-art solutions are substantially more vulnerable to forgetting on these complex deep reinforcement learning tasks.