mlsanityData to 14 Oct 2026

Essentials for Class Incremental Learning

arXiv 2102.09517 ↗First posted 18 Feb 2021cs.CV cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

Contemporary neural networks are limited in their ability to learn from evolving streams of training data. When trained sequentially on new or evolving tasks, their accuracy drops sharply, making them unsuitable for many real-world applications. In this work, we shed light on the causes of this well-known yet unsolved phenomenon - often referred to as catastrophic forgetting - in a class-incremental setup. We show that a combination of simple components and a loss that balances intra-task and inter-task learning can already resolve forgetting to the same extent as more complex measures proposed in literature. Moreover, we identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL. We show that performance is correlated with secondary class information (dark knowledge) learned by the model and it can be improved by an appropriate regularizer. With these lessons learned, class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

23 of 36 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2016Learning without Forgettingcited by 2,496
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2019Large Scale Incremental Learningcited by 732
  11. 2019Three scenarios for continual learningcited by 614
  12. 2018End-to-End Incremental Learningcited by 597
11 more
  1. 2018Born Again Neural Networkscited by 546
  2. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  3. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  4. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  5. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  6. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 221
  7. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  8. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  9. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  10. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  11. 2017Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networkscited by 59

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 17; 2023: 10; 2024: 15; 2025: 8; 2026: 2.

57 papers.

Sort:

By area:

  1. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  2. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  3. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  4. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 65
  5. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  6. 2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49
  7. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  8. 2021Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgettingcited by 33
  9. 2022Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentationcited by 28
  10. 2022SATS: Self-Attention Transfer for Continual Semantic Segmentationcited by 25
  11. 2023Dealing with Cross-Task Class Discrimination in Online Continual Learningcited by 24
  12. 2024Uncertainty-Calibrated Test-Time Model Adaptation without Forgettingcited by 20
38 more
  1. 2022Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learningcited by 16
  2. 2022Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learningcited by 15
  3. 2023LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platformscited by 11
  4. 2024Random Representations Outperform Online Continually Learned Representationscited by 11
  5. 2022A Memory Transformer Network for Incremental Learningcited by 9
  6. 2022CNLL: A Semi-supervised Approach For Continual Noisy Label Learningcited by 8
  7. 2024Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Modelscited by 8
  8. 2024Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perceptioncited by 8
  9. 2021Revisiting Catastrophic Forgetting in Class Incremental Learningcited by 6
  10. 2025CSTA: Spatial-Temporal Causal Adaptive Learning for Exemplar-Free Video Class-Incremental Learningcited by 5
  11. 2021Concept Representation Learning with Contrastive Self-Supervised Learningcited by 4
  12. 2022Diagnosing Batch Normalization in Class Incremental Learningcited by 4
  13. 2022Continual Learning for Pose-Agnostic Object Recognition in 3D Point Cloudscited by 4
  14. 2022Schedule-Robust Online Continual Learningcited by 4
  15. 2022Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learningcited by 3
  16. 2023FDCNet: Feature Drift Compensation Network for Class-Incremental Weakly Supervised Object Localizationcited by 3
  17. 2024Balancing Continual Learning and Fine-tuning for Human Activity Recognitioncited by 3
  18. 2024Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognitioncited by 3
  19. 2024Data-Free Federated Class Incremental Learning with Diffusion-Based Generative Memorycited by 3
  20. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  21. 2023Are Labels Needed for Incremental Instance Learning?cited by 2
  22. 2023Preserving Locality in Vision Transformers for Class Incremental Learningcited by 2
  23. 2023torchosr -- a PyTorch extension package for Open Set Recognition models evaluation in Pythoncited by 2
  24. 2024Adapter-Enhanced Semantic Prompting for Continual Learningcited by 2
  25. 2025EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learningcited by 2
  26. 2025FIESTA: Fisher Information-based Efficient Selective Test-time Adaptationcited by 2
  27. 2023Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Expertscited by 1
  28. 2024Hierarchical Prompts for Rehearsal-free Continual Learningcited by 1
  29. 2024Improving Forward Compatibility in Class Incremental Learning by Increasing Representation Rank and Feature Richnesscited by 1
  30. 2024Densely Distilling Cumulative Knowledge for Continual Learningcited by 1
  31. 2022Incremental Prototype Tuning for Class Incremental Learningcited by 0
  32. 2022Improving Feature Generalizability with Multitask Learning in Class Incremental Learningcited by 0
  33. 2022Neural Dependencies Emerging from Learning Massive Categoriescited by 0
  34. 2023Applications of Sequential Learning for Medical Image Classificationcited by 0
  35. 2023Negotiated Representations to Prevent Forgetting in Machine Learning Applicationscited by 0
  36. 2024HateDebias: On the Diversity and Variability of Hate Speech Debiasingcited by 0
  37. 2024Continual Learning of Nonlinear Independent Representationscited by 0
  38. 2024A Framework for Real-Time Volcano-Seismic Event Recognition Based on Multi-Station Seismograms and Semantic Segmentation Modelscited by 0

50 most cited of 57 All 57 (JSON)