The Sequential Price of Continual Learning
Not on the map. Why?
Abstract (arXiv)
Sequential task updates are fundamental to continual learning, but their recency bias can impose a lasting performance cost. We study this cost in an overparameterized linear-regression model with i.i.d. task sampling. We prove that distribution-level forgetting and population loss converge to the same stationary limit. We quantify the additional loss incurred by sequential exact fitting, or the sequential price. In more homogeneous task geometries, it equals the intrinsic loss asymptotically attained by joint training, making the total loss twice as large. We further analyze fixed-strength elastic weight consolidation (EWC) under general task curvatures and characterize its stationary sequential price at every regularization strength. Under strong regularization, the price decays inversely with EWC strength while the mean-square coupling horizon grows proportionally. Experiments on Jester and Rotated MNIST support the predicted sequential price and its reduction by EWC, with quantitative agreement on real-world tasks satisfying the theory's assumptions and qualitative agreement under nonlinear finite-step training.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 66 references that are in the map. First posted on arXiv 30 Aug 2026.
Builds on
50 of 66 references.
References · 50
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
- 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2018Progress & Compress: A scalable framework for continual learningcited by 520
- 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
- 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
- 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
38 more
- 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
- 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 112
- 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
- 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 88
- 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
- 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
- 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
- 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
- 2023Continual Learning in Linear Classification on Separable Datacited by 34
- 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
- 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 30
- 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
- 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
- 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
- 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
- 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
- 2023Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also Howcited by 16
- 2021Lifelong Learning with Sketched Structural Regularizationcited by 15
- 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15
- 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
- 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
- 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
- 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
- 2025Convergence and Implicit Bias of Gradient Descent on Continual Linear Classificationcited by 9
- 2025Bayesian continual learning and forgetting in neural networkscited by 9
- 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
- 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
- 2025Replay Can Provably Increase Forgettingcited by 5
- 2025Analysis of Overparameterization in Continual Learning under a Linear Modelcited by 4
- 2026Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptationcited by 4
- 2023IF2Net: Innately Forgetting-Free Networks for Continual Learningcited by 3
- 2025Are Greedy Task Orderings Better Than Random in Continual Linear Regression?cited by 3
- 2026Optimal L2 Regularization in High-dimensional Continual Linear Regressioncited by 3
- 2015Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysiscited by 2
- 2025Parabolic Continual Learningcited by 2
- 2025Supervised Learning with Evolving Tasks and Performance Guaranteescited by 1
- 2025Compact Memory for Continual Logistic Regressioncited by 1
- 2026Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgettingcited by 1
Built on by
Not cited yet.