# Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy

arXiv: https://arxiv.org/abs/2312.10549 · HTML: https://mlsanity.com/paper/2312.10549/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 120 of its 200 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 30 of 120)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)

## Built on by (most cited first, 24 of 24)

- [TxGemma: Efficient and Agentic LLMs for Therapeutics](https://mlsanity.com/paper/2504.06196/) (2025)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [FadeMem: Biologically-Inspired Forgetting for Efficient Agent Memory](https://mlsanity.com/paper/2601.18642/) (2026)
- [How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?](https://mlsanity.com/paper/2502.14502/) (2025)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation](https://mlsanity.com/paper/2501.06481/) (2025)
- [DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Robotics](https://mlsanity.com/paper/2507.04661/) (2025)
- [ExpReS-VLA: Specializing Vision-Language-Action Models Through Experience Replay and Retrieval](https://mlsanity.com/paper/2511.06202/) (2025)
- [Fairness and Robustness in Machine Unlearning](https://mlsanity.com/paper/2504.13610/) (2025)
- [MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation](https://mlsanity.com/paper/2508.11032/) (2025)
- [Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression](https://mlsanity.com/paper/2406.11354/) (2024)
- [REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Models](https://mlsanity.com/paper/2410.12890/) (2024)
- [Cattle-CLIP: A Multimodal Framework for Cattle Behaviour Recognition from Video](https://mlsanity.com/paper/2510.09203/) (2025)
- [Catastrophic Forgetting in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2511.12828/) (2025)
- [Fast Evaluation of DNN for Past Dataset in Incremental Learning](https://mlsanity.com/paper/2405.06296/) (2024)
- [Enabling On-Device Learning via Experience Replay with Efficient Dataset Condensation](https://mlsanity.com/paper/2405.16113/) (2024)
- [Skewed Neuronal Heterogeneity Enhances Efficiency On Various Computing Systems](https://mlsanity.com/paper/2412.05126/) (2024)
- [Leveraging Confident Image Regions for Source-Free Domain-Adaptive Object Detection](https://mlsanity.com/paper/2501.10081/) (2025)
- [Optimisation Is Not What You Need](https://mlsanity.com/paper/2507.03045/) (2025)
- [Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling](https://mlsanity.com/paper/2507.08736/) (2025)
- [Efficient Unlearning through Maximizing Relearning Convergence Delay](https://mlsanity.com/paper/2604.09391/) (2026)
- [Improving LLM Code Generation via Requirement-Aware Curriculum Reinforcement Learning](https://mlsanity.com/paper/2605.00433/) (2026)
- [A Continual Validation, Updating, and Decision-Making Framework for Self-Adaptive Digital Twins via Robust Model Predictive Control: A Case Study in Additive Manufacturing](https://mlsanity.com/paper/2607.18164/) (2026)
- [Continual Reinforcement Learning with Neuroevolution](https://mlsanity.com/paper/2610.01583/) (2026)

## Abstract (arXiv)

Deep Learning models have achieved remarkable performance in tasks such as image classification or generation, often surpassing human accuracy. However, they can struggle to learn new tasks and update their knowledge without access to previous data, leading to a significant loss of accuracy known as Catastrophic Forgetting (CF). This phenomenon was first observed by McCloskey and Cohen in 1989 and remains an active research topic. Incremental learning without forgetting is widely recognized as a crucial aspect in building better AI systems, as it allows models to adapt to new tasks without losing the ability to perform previously learned ones. This article surveys recent studies that tackle CF in modern Deep Learning models that use gradient descent as their learning algorithm. Although several solutions have been proposed, a definitive solution or consensus on assessing CF is yet to be established. The article provides a comprehensive review of recent solutions, proposes a taxonomy to organize them, and identifies research gaps in this area.