# Continual Learning and Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2403.05175 · HTML: https://mlsanity.com/paper/2403.05175/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 119 of its 177 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 30 of 119)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)

## Built on by (most cited first, 30 of 77)

- [LeanAgent: Lifelong Learning for Formal Theorem Proving](https://mlsanity.com/paper/2410.06209/) (2024)
- [Rehearsal-Free Modular and Compositional Continual Learning for Language Models](https://mlsanity.com/paper/2404.00790/) (2024)
- [Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models](https://mlsanity.com/paper/2508.09874/) (2025)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecasting](https://mlsanity.com/paper/2410.12593/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansion](https://mlsanity.com/paper/2502.12598/) (2025)
- [Optimal Rates in Continual Linear Regression via Increasing Regularization](https://mlsanity.com/paper/2506.06501/) (2025)
- [TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learning](https://mlsanity.com/paper/2512.20312/) (2025)
- [Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents](https://mlsanity.com/paper/2605.17830/) (2026)
- [Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation](https://mlsanity.com/paper/2501.15377/) (2025)
- [UniCoRN: Latent Diffusion-based Unified Controllable Image Restoration Network across Multiple Degradations](https://mlsanity.com/paper/2503.15868/) (2025)
- [SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2409.00055/) (2024)
- [DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Experts](https://mlsanity.com/paper/2508.07842/) (2025)
- [Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training](https://mlsanity.com/paper/2605.09608/) (2026)
- [MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning](https://mlsanity.com/paper/2603.09892/) (2026)
- [Replay4NCL: An Efficient Memory Replay-based Methodology for Neuromorphic Continual Learning in Embedded AI Systems](https://mlsanity.com/paper/2503.17061/) (2025)
- [Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning](https://mlsanity.com/paper/2503.21419/) (2025)
- [We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong](https://mlsanity.com/paper/2509.22510/) (2025)
- [CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion](https://mlsanity.com/paper/2405.09637/) (2024)
- [U-TELL: Unsupervised Task Expert Lifelong Learning](https://mlsanity.com/paper/2405.14623/) (2024)
- [Lifelong Learning of Video Diffusion Models From a Single Video Stream](https://mlsanity.com/paper/2406.04814/) (2024)
- [Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibility](https://mlsanity.com/paper/2502.13935/) (2025)
- [Exploring Incremental Unlearning: Techniques, Challenges, and Future Directions](https://mlsanity.com/paper/2502.16708/) (2025)
- [LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2503.21227/) (2025)
- [Replay to Remember (R2R): An Efficient Uncertainty-driven Unsupervised Continual Learning Framework Using Generative Replay](https://mlsanity.com/paper/2505.04787/) (2025)
- [Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldone](https://mlsanity.com/paper/2505.20113/) (2025)
- [Continual Learning with Columnar Spiking Neural Networks](https://mlsanity.com/paper/2506.17169/) (2025)
- [Spatial Audio Motion Understanding and Reasoning](https://mlsanity.com/paper/2509.14666/) (2025)

## Abstract (arXiv)

This book chapter delves into the dynamics of continual learning, which is the process of incrementally learning from a non-stationary stream of data. Although continual learning is a natural skill for the human brain, it is very challenging for artificial neural networks. An important reason is that, when learning something new, these networks tend to quickly and drastically forget what they had learned before, a phenomenon known as catastrophic forgetting. Especially in the last decade, continual learning has become an extensively studied topic in deep learning. This book chapter reviews the insights that this field has generated.