mlsanityData to 7 Oct 2026

A Theoretical Study on Solving Continual Learning

arXiv 2211.02633 ↗First posted 4 Nov 2022cs.LG cs.AI cs.CVcited by 78

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) learns a sequence of tasks incrementally. There are two popular CL settings, class incremental learning (CIL) and task incremental learning (TIL). A major challenge of CL is catastrophic forgetting (CF). While a number of techniques are already available to effectively overcome CF for TIL, CIL remains to be highly challenging. So far, little theoretical study has been done to provide a principled guidance on how to solve the CIL problem. This paper performs such a study. It first shows that probabilistically, the CIL problem can be decomposed into two sub-problems: Within-task Prediction (WP) and Task-id Prediction (TP). It further proves that TP is correlated with out-of-distribution (OOD) detection, which connects CIL and OOD detection. The key conclusion of this study is that regardless of whether WP and TP or OOD detection are defined explicitly or implicitly by a CIL algorithm, good WP and good TP or OOD detection are necessary and sufficient for good CIL performances. Additionally, TIL is simply WP. Based on the theoretical result, new CIL methods are also designed, which outperform strong baselines in both CIL and TIL settings by a large margin.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 68 of its 86 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

68 of 86 references.

References · 68

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2016Learning without Forgettingcited by 2,465
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
38 more
  1. 2016Progressive Neural Networkscited by 1,332
  2. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  3. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2017Continual Learning with Deep Generative Replaycited by 1,152
  6. 2018Experience Replay for Continual Learningcited by 1,074
  7. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  8. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  10. 2019Large Scale Incremental Learningcited by 727
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  12. 2019Three scenarios for continual learningcited by 610
  13. 2018End-to-End Incremental Learningcited by 596
  14. 2017Large Batch Training of Convolutional Networkscited by 529
  15. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  16. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  17. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  18. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  19. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  20. 2018Recent Advances in Open Set Recognition: A Surveycited by 309
  21. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  22. 2019Continual learning with hypernetworkscited by 267
  23. 2018Learning without Memorizingcited by 249
  24. 2021Co2L: Contrastive Continual Learningcited by 243
  25. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  26. 2020Supermasks in Superpositioncited by 223
  27. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  28. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  29. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  30. 2018Reinforced Continual Learningcited by 186
  31. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 181
  32. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  33. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  34. 2020Adaptive Aggregation Networks for Class-Incremental Learningcited by 149
  35. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  36. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  37. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  38. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 24; 2024: 27; 2025: 18; 2026: 8.

78 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023Class-Incremental Learning: A Surveycited by 209
  4. 2023Continual Pre-training of Language Modelscited by 194
  5. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  6. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  7. 2024Continual Learning and Catastrophic Forgettingcited by 73
  8. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  9. 2023Continual Learning: Applications and the Road Forwardcited by 59
  10. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  11. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  12. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 32
38 more
  1. 2023Learnability and Algorithm for Continual Learningcited by 28
  2. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  3. 2023Dealing with Cross-Task Class Discrimination in Online Continual Learningcited by 24
  4. 2023ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithmscited by 22
  5. 2024OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learningcited by 20
  6. 2024HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuningcited by 20
  7. 2023Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learningcited by 18
  8. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  9. 2023Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also Howcited by 16
  10. 2023Prediction Error-based Classification for Class-Incremental Learningcited by 14
  11. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  12. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 13
  13. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  14. 2024Replay-and-Forget-Free Graph Class-Incremental Learning: A Task Profiling and Prompting Approachcited by 12
  15. 2023Class-Incremental Learning based on Label Generationcited by 11
  16. 2023Continual Evidential Deep Learning for Out-of-Distribution Detectioncited by 11
  17. 2024Integrating Present and Past in Unsupervised Continual Learningcited by 8
  18. 2025On the Discrimination and Consistency for Exemplar-Free Class Incremental Learningcited by 8
  19. 2025On-device edge learning for IoT data streams: a surveycited by 8
  20. 2023Partial Hypernetworks for Continual Learningcited by 7
  21. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  22. 2024PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learningcited by 7
  23. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  24. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  25. 2023Non-exemplar Class-incremental Learning by Random Auxiliary Classes Augmentation and Mixed Featurescited by 6
  26. 2024Federated Class-Incremental Learning with Hierarchical Generative Prototypescited by 6
  27. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  28. 2024Continual Learning Using Only Large Language Model Promptingcited by 5
  29. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  30. 2024IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learningcited by 4
  31. 2024CroMo-Mixup: Augmenting Cross-Model Representations for Continual Self-Supervised Learningcited by 4
  32. 2024Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelingscited by 4
  33. 2025Achieving Upper Bound Accuracy of Joint Training in Continual Learningcited by 4
  34. 2025H2ST: Hierarchical Two-Sample Tests for Continual Out-of-Distribution Detectioncited by 4
  35. 2025Continual Hyperbolic Learning of Instances and Classescited by 4
  36. 2023Efficient Expansion and Gradient Based Task Inference for Replay Free Incremental Learningcited by 3
  37. 2024A Parameter-efficient Language Extension Framework for Multilingual ASRcited by 3
  38. 2023Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learningcited by 2

50 most cited of 78 All 78 (JSON)