# Essentials for Class Incremental Learning

arXiv: https://arxiv.org/abs/2102.09517 · HTML: https://mlsanity.com/paper/2102.09517/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [SS-IL: Separated Softmax for Incremental Learning](https://mlsanity.com/paper/2003.13947/) (2020)
- [Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild](https://mlsanity.com/paper/1903.12648/) (2019)
- [Memory-Efficient Incremental Learning Through Feature Adaptation](https://mlsanity.com/paper/2004.00713/) (2020)
- [Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks](https://mlsanity.com/paper/1702.05870/) (2017)

## Built on by (most cited first, 30 of 57)

- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning](https://mlsanity.com/paper/2112.04731/) (2021)
- [A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials](https://mlsanity.com/paper/2205.05467/) (2022)
- [Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting](https://mlsanity.com/paper/2108.08165/) (2021)
- [Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.05941/) (2022)
- [SATS: Self-Attention Transfer for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.07667/) (2022)
- [Dealing with Cross-Task Class Discrimination in Online Continual Learning](https://mlsanity.com/paper/2305.14657/) (2023)
- [Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2403.11491/) (2024)
- [Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning](https://mlsanity.com/paper/2201.12559/) (2022)
- [Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning](https://mlsanity.com/paper/2212.14284/) (2022)
- [LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms](https://mlsanity.com/paper/2311.11420/) (2023)
- [Random Representations Outperform Online Continually Learned Representations](https://mlsanity.com/paper/2402.08823/) (2024)
- [A Memory Transformer Network for Incremental Learning](https://mlsanity.com/paper/2210.04485/) (2022)
- [CNLL: A Semi-supervised Approach For Continual Noisy Label Learning](https://mlsanity.com/paper/2204.09881/) (2022)
- [Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Models](https://mlsanity.com/paper/2406.09384/) (2024)
- [Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception](https://mlsanity.com/paper/2407.18145/) (2024)
- [Revisiting Catastrophic Forgetting in Class Incremental Learning](https://mlsanity.com/paper/2107.12308/) (2021)
- [CSTA: Spatial-Temporal Causal Adaptive Learning for Exemplar-Free Video Class-Incremental Learning](https://mlsanity.com/paper/2501.07236/) (2025)
- [Concept Representation Learning with Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2112.05677/) (2021)
- [Diagnosing Batch Normalization in Class Incremental Learning](https://mlsanity.com/paper/2202.08025/) (2022)
- [Continual Learning for Pose-Agnostic Object Recognition in 3D Point Clouds](https://mlsanity.com/paper/2209.04840/) (2022)
- [Schedule-Robust Online Continual Learning](https://mlsanity.com/paper/2210.05561/) (2022)
- [Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning](https://mlsanity.com/paper/2211.11174/) (2022)
- [FDCNet: Feature Drift Compensation Network for Class-Incremental Weakly Supervised Object Localization](https://mlsanity.com/paper/2309.09122/) (2023)
- [Balancing Continual Learning and Fine-tuning for Human Activity Recognition](https://mlsanity.com/paper/2401.02255/) (2024)
- [Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition](https://mlsanity.com/paper/2401.06287/) (2024)

## Abstract (arXiv)

Contemporary neural networks are limited in their ability to learn from evolving streams of training data. When trained sequentially on new or evolving tasks, their accuracy drops sharply, making them unsuitable for many real-world applications. In this work, we shed light on the causes of this well-known yet unsolved phenomenon - often referred to as catastrophic forgetting - in a class-incremental setup. We show that a combination of simple components and a loss that balances intra-task and inter-task learning can already resolve forgetting to the same extent as more complex measures proposed in literature. Moreover, we identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL. We show that performance is correlated with secondary class information (dark knowledge) learned by the model and it can be improved by an appropriate regularizer. With these lessons learned, class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.