mlsanityData to 7 Oct 2026

Memory-Statistics Tradeoff in Continual Learning with Structural Regularization

arXiv 2504.04039 ↗First posted 5 Apr 2025cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

We study the statistical performance of a continual learning problem with two linear regression tasks in a well-specified random design setting. We consider a structural regularization algorithm that incorporates a generalized ℓ2-regularization tailored to the Hessian of the previous task for mitigating catastrophic forgetting. We establish upper and lower bounds on the joint excess risk for this algorithm. Our analysis reveals a fundamental trade-off between memory complexity and statistical efficiency, where memory complexity is measured by the number of vectors needed to define the structural regularization. Specifically, increasing the number of vectors in structural regularization leads to a worse memory complexity but an improved excess risk, and vice versa. Furthermore, our theory suggests that naive continual learning without regularization suffers from catastrophic forgetting, while structural regularization mitigates this issue. Notably, structural regularization achieves comparable performance to joint training with access to both tasks simultaneously. These results highlight the critical role of curvature-aware regularization for continual learning.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

29 of 36 references.

References · 29

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  8. 2019Three scenarios for continual learningcited by 610
  9. 2019Benign Overfitting in Linear Regressioncited by 544
  10. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  11. 2021Gradient Projection Memory for Continual Learningcited by 338
  12. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
17 more
  1. 2020Benign overfitting in ridge regressioncited by 152
  2. 2024Continual Learning for Large Language Models: A Surveycited by 144
  3. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  4. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  5. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  6. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  7. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  8. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  9. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  10. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  11. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  12. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  13. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  14. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 25
  15. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  16. 2021Lifelong Learning with Sketched Structural Regularizationcited by 15
  17. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
  2. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  3. 2025Compact Memory for Continual Logistic Regressioncited by 1
  4. 2026Beyond Pooling: Matching for Robust Generalization under Data Heterogeneitycited by 0