mlsanityData to 7 Oct 2026

Fixed Design Analysis of Regularization-Based Continual Learning

arXiv 2303.10263 ↗First posted 17 Mar 2023cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

We consider a continual learning (CL) problem with two linear regression tasks in the fixed design setting, where the feature vectors are assumed fixed and the labels are assumed to be random variables. We consider an ℓ2-regularized CL algorithm, which computes an Ordinary Least Squares parameter to fit the first dataset, then computes another parameter that fits the second dataset under an ℓ2-regularization penalizing its deviation from the first parameter, and outputs the second parameter. For this algorithm, we provide tight bounds on the average risk over the two tasks. Our risk bounds reveal a provable trade-off between forgetting and intransigence of the ℓ2-regularized CL algorithm: with a large regularization parameter, the algorithm output forgets less information about the first task but is intransigent to extract new information from the second task; and vice versa. Our results suggest that catastrophic forgetting could happen for CL with dissimilar tasks (under a precise similarity measurement) and that a well-tuned ℓ2-regularization can partially mitigate this issue by introducing intransigence.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

31 of 41 references.

References · 31

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  6. 2017Continual Learning with Deep Generative Replaycited by 1,152
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  10. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  12. 2019Three scenarios for continual learningcited by 610
19 more
  1. 2019Benign Overfitting in Linear Regressioncited by 544
  2. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  3. 2021Gradient Projection Memory for Continual Learningcited by 338
  4. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  5. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  6. 2018Reinforced Continual Learningcited by 186
  7. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  8. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  9. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  10. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  11. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  12. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  13. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  14. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  15. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  16. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 25
  17. 2022Memory Bounds for Continual Learningcited by 18
  18. 2021Lifelong Learning with Sketched Structural Regularizationcited by 15
  19. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 12; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  2. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  3. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  4. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  5. 2024Provable Contrastive Continual Learningcited by 13
  6. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  7. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
  8. 2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 10
  9. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  10. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  11. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  12. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 4
12 more
  1. 2025Replay Can Provably Increase Forgettingcited by 4
  2. 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
  3. 2024Statistical curriculum learning: An elimination algorithm achieving an oracle riskcited by 1
  4. 2025Compact Memory for Continual Logistic Regressioncited by 1
  5. 2025High-dimensional Asymptotics of Generalization Performance in Continual Ridge Regressioncited by 0
  6. 2025Efficient Online Continual Learning in Sensor-Based Human Activity Recognitioncited by 0
  7. 2025Unified Class and Domain Incremental Learning with Mixture of Experts for Indoor Localizationcited by 0
  8. 2026Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacitiescited by 0
  9. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 0
  10. 2026Theory of Continual Learning Against Data Poisoning Attackscited by 0
  11. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  12. 2026Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learningcited by 0