mlsanityData to 14 Oct 2026

Three scenarios for continual learning

arXiv 1904.07734 ↗First posted 15 Apr 2019cs.LG cs.AI cs.CVcited by 614

Not on the map. Why?

Abstract (arXiv)

Standard artificial neural networks suffer from the well-known issue of catastrophic forgetting, making continual or lifelong learning difficult for machine learning. In recent years, numerous methods have been proposed for continual learning, but due to differences in evaluation protocols it is difficult to directly compare their performance. To enable more structured comparisons, we describe three continual learning scenarios based on whether at test time task identity is provided and--in case it is not--whether it must be inferred. Any sequence of well-defined tasks can be performed according to each scenario. Using the split and permuted MNIST task protocols, for each scenario we carry out an extensive comparison of recently proposed continual learning methods. We demonstrate substantial differences between the three scenarios in terms of difficulty and in terms of how efficient different methods are. In particular, when task identity must be inferred (i.e., class incremental learning), we find that regularization-based approaches (e.g., elastic weight consolidation) fail and that replaying representations of previous experiences seems required for solving this scenario.

Key facts. Cited by 614 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 93; 2022: 113; 2023: 134; 2024: 135; 2025: 88; 2026: 51.

614 papers.

Sort:

By area:

  1. 2021Learning to Prompt for Continual Learningcited by 944
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  4. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  5. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  6. 2021Co2L: Contrastive Continual Learningcited by 244
  7. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 219
  8. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  9. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  10. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  11. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  12. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
38 more
  1. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  2. 2022Continual Learning and Private Unlearningcited by 130
  3. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  4. 2022FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learningcited by 121
  5. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  6. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  7. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 115
  8. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  9. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 112
  10. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  11. 2021Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learningcited by 96
  12. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  13. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 93
  14. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  15. 2023FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learningcited by 87
  16. 2021On Learning the Geodesic Path for Incremental Learningcited by 85
  17. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  18. 2023Modular Deep Learningcited by 83
  19. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  20. 2022Memory Replay with Data Compression for Continual Learningcited by 82
  21. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
  22. 2022A Theoretical Study on Solving Continual Learningcited by 80
  23. 2021Rehearsal revealed: The limits and merits of revisiting samples in continual learningcited by 76
  24. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  25. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  26. 2021SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learningcited by 71
  27. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  28. 2025End-to-End Test-Time Training for Long Contextcited by 62
  29. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  30. 2021Essentials for Class Incremental Learningcited by 57
  31. 2021Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworkscited by 56
  32. 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 56
  33. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  34. 2023Better Generative Replay for Continual Federated Learningcited by 55
  35. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 54
  36. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  37. 2022An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditionscited by 54
  38. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 54

50 most cited of 614 All 614 (JSON)