# Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks

arXiv: https://arxiv.org/abs/1802.03875 · HTML: https://mlsanity.com/paper/1802.03875/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2018.

## Built on by (most cited first, 23 of 23)

- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Continual Sequence Generation with Adaptive Compositional Modules](https://mlsanity.com/paper/2203.10652/) (2022)
- [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021)
- [AirLoop: Lifelong Loop Closure Detection](https://mlsanity.com/paper/2109.08975/) (2021)
- [The Importance of Robust Features in Mitigating Catastrophic Forgetting](https://mlsanity.com/paper/2306.17091/) (2023)
- [CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering](https://mlsanity.com/paper/2211.10567/) (2022)
- [CFA: Constraint-based Finetuning Approach for Generalized Few-Shot Object Detection](https://mlsanity.com/paper/2204.05220/) (2022)
- [Overcoming Catastrophic Forgetting in Tabular Data Classification: A Pseudorehearsal-based approach](https://mlsanity.com/paper/2407.09039/) (2024)
- [Tuned Compositional Feature Replays for Efficient Stream Learning](https://mlsanity.com/paper/2104.02206/) (2021)
- [Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning](https://mlsanity.com/paper/2411.06764/) (2024)
- [Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning](https://mlsanity.com/paper/2211.11174/) (2022)
- [Integral Continual Learning Along the Tangent Vector Field of Tasks](https://mlsanity.com/paper/2211.13108/) (2022)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [Integrating Curricula with Replays: Its Effects on Continual Learning](https://mlsanity.com/paper/2307.05747/) (2023)
- [I3DOD: Towards Incremental 3D Object Detection via Prompting](https://mlsanity.com/paper/2308.12512/) (2023)
- [Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization](https://mlsanity.com/paper/2312.16731/) (2023)
- [FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization](https://mlsanity.com/paper/2410.00508/) (2024)
- [SHARP: Sparsity and Hidden Activation RePlay for Neuro-Inspired Continual Learning](https://mlsanity.com/paper/2305.18563/) (2023)
- [ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning](https://mlsanity.com/paper/2103.12216/) (2021)
- [Brain-Inspired Continual Learning-Robust Feature Distillation and Re-Consolidation for Class Incremental Learning](https://mlsanity.com/paper/2404.14588/) (2024)
- [Continual Learning in Machine Speech Chain Using Gradient Episodic Memory](https://mlsanity.com/paper/2411.18320/) (2024)

## Abstract (arXiv)

In general, neural networks are not currently capable of learning tasks in a sequential fashion. When a novel, unrelated task is learnt by a neural network, it substantially forgets how to solve previously learnt tasks. One of the original solutions to this problem is pseudo-rehearsal, which involves learning the new task while rehearsing generated items representative of the previous task/s. This is very effective for simple tasks. However, pseudo-rehearsal has not yet been successfully applied to very complex tasks because in these tasks it is difficult to generate representative items. We accomplish pseudo-rehearsal by using a Generative Adversarial Network to generate items so that our deep network can learn to sequentially classify the CIFAR-10, SVHN and MNIST datasets. After training on all tasks, our network loses only 1.67% absolute accuracy on CIFAR-10 and gains 0.24% absolute accuracy on SVHN. Our model's performance is a substantial improvement compared to the current state of the art solution.