# A continual learning survey: Defying forgetting in classification tasks

arXiv: https://arxiv.org/abs/1909.08383 · HTML: https://mlsanity.com/paper/1909.08383/

## Key facts

- Cited by 118 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 30 of 118)

- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2211.13131/) (2022)
- [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Continual Learning of Natural Language Processing Tasks: A Survey](https://mlsanity.com/paper/2211.12701/) (2022)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [RMM: Reinforced Memory Management for Class-Incremental Learning](https://mlsanity.com/paper/2301.05792/) (2023)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [Self-Supervised Training Enhances Online Continual Learning](https://mlsanity.com/paper/2103.14010/) (2021)
- [Efficient Feature Transformations for Discriminative and Generative Continual Learning](https://mlsanity.com/paper/2103.13558/) (2021)
- [Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.03271/) (2021)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [Continuum: Simple Management of Complex Continual Learning Scenarios](https://mlsanity.com/paper/2102.06253/) (2021)
- [CITB: A Benchmark for Continual Instruction Tuning](https://mlsanity.com/paper/2310.14510/) (2023)
- [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.02714/) (2021)
- [Online Continual Learning Without the Storage Constraint](https://mlsanity.com/paper/2305.09253/) (2023)
- [Domain Generalization through Meta-Learning: A Survey](https://mlsanity.com/paper/2404.02785/) (2024)
- [Kernel Continual Learning](https://mlsanity.com/paper/2107.05757/) (2021)
- [Multi-Domain Incremental Learning for Semantic Segmentation](https://mlsanity.com/paper/2110.12205/) (2021)
- [Revisiting a kNN-based Image Classification System with High-capacity Storage](https://mlsanity.com/paper/2204.01186/) (2022)
- [Exploring Continual Learning for Code Generation Models](https://mlsanity.com/paper/2307.02435/) (2023)
- [Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning](https://mlsanity.com/paper/2309.01289/) (2023)
- [Challenging Common Assumptions about Catastrophic Forgetting](https://mlsanity.com/paper/2207.04543/) (2022)
- [Prototype-Guided Memory Replay for Continual Learning](https://mlsanity.com/paper/2108.12641/) (2021)

## Abstract (arXiv)

Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting. Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch. We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries. Our main contributions concern 1) a taxonomy and extensive overview of the state-of-the-art, 2) a novel framework to continually determine the stability-plasticity trade-off of the continual learner, 3) a comprehensive experimental comparison of 11 state-of-the-art continual learning methods and 4 baselines. We empirically scrutinize method strengths and weaknesses on three benchmarks, considering Tiny Imagenet and large-scale unbalanced iNaturalist and a sequence of recognition datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time, and storage.