# Toward Understanding Catastrophic Forgetting in Continual Learning

arXiv: https://arxiv.org/abs/1908.01091 · HTML: https://mlsanity.com/paper/1908.01091/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2019.

## Built on by (most cited first, 30 of 68)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Probing Representation Forgetting in Supervised and Unsupervised Continual Learning](https://mlsanity.com/paper/2203.13381/) (2022)
- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [A Review of Deep Transfer Learning and Recent Advancements](https://mlsanity.com/paper/2201.09679/) (2022)
- [A Closer Look at Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2203.17269/) (2022)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Mixture of Experts Meets Prompt-Based Continual Learning](https://mlsanity.com/paper/2405.14124/) (2024)
- [The Effect of Task Ordering in Continual Learning](https://mlsanity.com/paper/2205.13323/) (2022)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Wasserstein Task Embedding for Measuring Task Similarities](https://mlsanity.com/paper/2208.11726/) (2022)
- [DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers](https://mlsanity.com/paper/2505.23694/) (2025)
- [iKAN: Global Incremental Learning with KAN for Human Activity Recognition Across Heterogeneous Datasets](https://mlsanity.com/paper/2406.01646/) (2024)
- [Towards Effective and Efficient Continual Pre-training of Large Language Models](https://mlsanity.com/paper/2407.18743/) (2024)
- [Continual Unlearning for Foundational Text-to-Image Models without Generalization Erosion](https://mlsanity.com/paper/2503.13769/) (2025)
- [Through the Valley: Path to Effective Long CoT Training for Small Language Models](https://mlsanity.com/paper/2506.07712/) (2025)
- [EVCL: Elastic Variational Continual Learning with Weight Consolidation](https://mlsanity.com/paper/2406.15972/) (2024)
- [Theoretical Understanding of the Information Flow on Continual Learning Performance](https://mlsanity.com/paper/2204.12010/) (2022)
- [Predicting the Susceptibility of Examples to Catastrophic Forgetting](https://mlsanity.com/paper/2406.09935/) (2024)
- [L3DMC: Lifelong Learning using Distillation via Mixed-Curvature Space](https://mlsanity.com/paper/2307.16459/) (2023)
- [Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors](https://mlsanity.com/paper/2505.20680/) (2025)
- [On robustness of generative representations against catastrophic forgetting](https://mlsanity.com/paper/2109.01844/) (2021)
- [Dendritic Self-Organizing Maps for Continual Learning](https://mlsanity.com/paper/2110.13611/) (2021)
- [Measuring and Mitigating Interference in Reinforcement Learning](https://mlsanity.com/paper/2307.04887/) (2023)
- [Continual Learning Improves Zero-Shot Action Recognition](https://mlsanity.com/paper/2410.10497/) (2024)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts](https://mlsanity.com/paper/2506.21035/) (2025)
- [Generalization Bounds for Deep Transfer Learning Using Majority Predictor Accuracy](https://mlsanity.com/paper/2209.05709/) (2022)

## Abstract (arXiv)

We study the relationship between catastrophic forgetting and properties of task sequences. In particular, given a sequence of tasks, we would like to understand which properties of this sequence influence the error rates of continual learning algorithms trained on the sequence. To this end, we propose a new procedure that makes use of recent developments in task space modeling as well as correlation analysis to specify and analyze the properties we are interested in. As an application, we apply our procedure to study two properties of a task sequence: (1) total complexity and (2) sequential heterogeneity. We show that error rates are strongly and positively correlated to a task sequence's total complexity for some state-of-the-art algorithms. We also show that, surprisingly, the error rates have no or even negative correlations in some cases to sequential heterogeneity. Our findings suggest directions for improving continual learning benchmarks and methods.