# Revisiting Catastrophic Forgetting in Class Incremental Learning

arXiv: https://arxiv.org/abs/2107.12308 · HTML: https://mlsanity.com/paper/2107.12308/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [Incremental Learning In Online Scenario](https://mlsanity.com/paper/2003.13191/) (2020)
- [Essentials for Class Incremental Learning](https://mlsanity.com/paper/2102.09517/) (2021)
- [Memory Efficient Class-Incremental Learning for Image Classification](https://mlsanity.com/paper/2008.01411/) (2020)
- [Regularization Shortcomings for Continual Learning](https://mlsanity.com/paper/1912.03049/) (2019)
- [Pseudo-Rehearsal for Continual Learning with Normalizing Flows](https://mlsanity.com/paper/2007.02443/) (2020)
- [Unifying Regularisation Methods for Continual Learning](https://mlsanity.com/paper/2006.06357/) (2020)

## Built on by (most cited first, 6 of 6)

- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Continual Vision-Language Representation Learning with Off-Diagonal Information](https://mlsanity.com/paper/2305.07437/) (2023)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion](https://mlsanity.com/paper/2308.02552/) (2023)
- [Task-Agnostic Guided Feature Expansion for Class-Incremental Learning](https://mlsanity.com/paper/2503.00823/) (2025)
- [Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants](https://mlsanity.com/paper/2308.01746/) (2023)

## Abstract (arXiv)

Although the concept of catastrophic forgetting is straightforward, there is a lack of study on its causes. In this paper, we systematically explore and reveal three causes for catastrophic forgetting in Class Incremental Learning(CIL). From the perspective of representation learning,(i) intra-phase forgetting happens when the learner fails to correctly align the same-phase data as training proceeds and (ii) inter-phase confusion happens when the learner confuses the current-phase data with the previous-phase. From the task-specific point of view, the CIL model suffers from the problem of (iii) classifier deviation. After investigating existing strategies, we observe that there is a lack of study on how to prevent the inter-phase confusion. To initiate the research on this specific issue, we propose a simple yet effective framework, Contrastive Class Concentration for CIL (C4IL). Our framework leverages the class concentration effect of contrastive learning, yielding a representation distribution with better intra-class compactibility and inter-class separability. Empirically, we observe that C4IL significantly lowers the probability of inter-phase confusion and as a result improves the performance on multiple CIL settings of multiple datasets.