mlsanityData to 7 Oct 2026

Global Convergence of Continual Learning on Non-IID Data

arXiv 2503.18511 ↗First posted 24 Mar 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Continual learning, which aims to learn multiple tasks sequentially, has gained extensive attention. However, most existing work focuses on empirical studies, and the theoretical aspect remains under-explored. Recently, a few investigations have considered the theory of continual learning only for linear regressions, establishes the results based on the strict independent and identically distributed (i.i.d.) assumption and the persistent excitation on the feature data that may be difficult to verify or guarantee in practice. To overcome this fundamental limitation, in this paper, we provide a general and comprehensive theoretical analysis for continual learning of regression models. By utilizing the stochastic Lyapunov function and martingale estimation techniques, we establish the almost sure convergence results of continual learning under a general data condition for the first time. Additionally, without any excitation condition imposed on the data, the convergence rates for the forgetting and regret metrics are provided.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

47 of 78 references.

References · 47

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2016Learning without Forgettingcited by 2,465
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2016Progressive Neural Networkscited by 1,332
  6. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  7. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  8. 2021Learning to Prompt for Continual Learningcited by 938
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  12. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
35 more
  1. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  2. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  3. 2023Continual Pre-training of Language Modelscited by 194
  4. 2018Reinforced Continual Learningcited by 186
  5. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  6. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  7. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 158
  8. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  9. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  10. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  11. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  12. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  13. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  14. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  15. 2020Gradient-based Editing of Memory Examples for Online Task-free Continual Learningcited by 62
  16. 2021Few-Shot Lifelong Learningcited by 58
  17. 2022ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protectioncited by 56
  18. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  19. 2018From Word to Sense Embeddings: A Survey on Vector Representations of Meaningcited by 49
  20. 2023A Survey on Few-Shot Class-Incremental Learningcited by 48
  21. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  22. 2022Long-Tailed Class Incremental Learningcited by 36
  23. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  24. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  25. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  26. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  27. 2023Learnability and Algorithm for Continual Learningcited by 28
  28. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  29. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26
  30. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  31. 2020Computationally and Statistically Efficient Truncated Regressioncited by 23
  32. 2020ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendationcited by 22
  33. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  34. 2024PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learningcited by 19
  35. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 3
  2. 2026Continual Learning for non-stationary regression via Memory-Efficient Replaycited by 0
  3. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 0