mlsanityData to 7 Oct 2026

A Statistical Theory of Regularization-Based Continual Learning

arXiv 2406.06213 ↗First posted 10 Jun 2024cs.LG cs.AI stat.APcited by 32

Not on the map. Why?

Abstract (arXiv)

We provide a statistical analysis of regularization-based continual learning on a sequence of linear regression tasks, with emphasis on how different regularization terms affect the model performance. We first derive the convergence rate for the oracle estimator obtained as if all data were available simultaneously. Next, we consider a family of generalized ℓ2-regularization algorithms indexed by matrix-valued hyperparameters, which includes the minimum norm estimator and continual ridge regression as special cases. As more tasks are introduced, we derive an iterative update formula for the estimation error of generalized ℓ2-regularized estimators, from which we determine the hyperparameters resulting in the optimal algorithm. Interestingly, the choice of hyperparameters can effectively balance the trade-off between forward and backward knowledge transfer and adjust for data heterogeneity. Moreover, the estimation error of the optimal algorithm is derived explicitly, which is of the same order as that of the oracle estimator. In contrast, our lower bounds for the minimum norm estimator and continual ridge regression show their suboptimality. A byproduct of our theoretical analysis is the equivalence between early stopping and generalized ℓ2-regularization in continual learning, which may be of independent interest. Finally, we conduct experiments to complement our theory.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

35 of 39 references.

References · 35

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  5. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  8. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  9. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  10. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  11. 2021Gradient Projection Memory for Continual Learningcited by 338
  12. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
23 more
  1. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  2. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  3. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  4. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  5. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  6. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  7. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  8. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  9. 2020Gradient-based Editing of Memory Examples for Online Task-free Continual Learningcited by 62
  10. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 54
  11. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  12. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  13. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  14. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  15. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 25
  16. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  17. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  18. 2022Memory Bounds for Continual Learningcited by 18
  19. 2021Provable Lifelong Learning of Representationscited by 15
  20. 2022Provable and Efficient Continual Representation Learningcited by 14
  21. 2024Provable Contrastive Continual Learningcited by 13
  22. 2021GROWN: GRow Only When Necessary for Continual Learningcited by 11
  23. 2022Theoretical Understanding of the Information Flow on Continual Learning Performancecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 15.

32 papers.

Sort:

By area:

  1. 2025Continual Multimodal Contrastive Learningcited by 19
  2. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  3. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
  4. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  5. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  6. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 8
  7. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  8. 2025Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learningcited by 7
  9. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  10. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 4
  11. 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
  12. 2025Cross-Domain Continual Learning for Edge Intelligence in Wireless ISAC Networkscited by 2
20 more
  1. 2025Analytic Subspace Routing: How Recursive Least Squares Works in Continual Learning of Large Language Modelcited by 2
  2. 2026Information-Theoretic Constraints for Continual Vision-Language-Action Alignmentcited by 2
  3. 2024Algorithm Design for Continual Learning in IoT Networkscited by 1
  4. 2025T2I-ConBench: Text-to-Image Benchmark for Continual Post-trainingcited by 1
  5. 2026Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspectivecited by 1
  6. 2025Statistical Theory of Multi-stage Newton Iteration Algorithm for Online Continual Learningcited by 0
  7. 2025High-dimensional Asymptotics of Generalization Performance in Continual Ridge Regressioncited by 0
  8. 2026Affect and Effect: Limitations of regularisation-based continual learning in EEG-based emotion classificationcited by 0
  9. 2026Continual Learning for non-stationary regression via Memory-Efficient Replaycited by 0
  10. 2026Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacitiescited by 0
  11. 2026Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learningcited by 0
  12. 2026Unsupervised Continual Learning for Amortized Bayesian Inferencecited by 0
  13. 2026From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learningcited by 0
  14. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 0
  15. 2026Scene-Adaptive Continual Learning for CSI-based Human Activity Recognition with Mixture of Expertscited by 0
  16. 2026The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learningcited by 0
  17. 2026Theory of Continual Learning Against Data Poisoning Attackscited by 0
  18. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  19. 2026Optimal Training-Time Scaling in Gradual Adaptationcited by 0
  20. 2026The Sequential Price of Continual Learningcited by 0