mlsanityData to 14 Oct 2026

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

arXiv 2604.13460 ↗First posted 15 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A central challenge in continual learning is forgetting: the loss of performance on previously learned tasks after learning new ones. Prior theory has analyzed forgetting under random orderings of fixed task collections in overparameterized linear regression. We shift the focus from task order to task distribution, asking how its structure determines forgetting. In the linear setting with a shared solution, i.i.d. task sampling, and sequential exact fitting, we derive an exact operator identity expressing historical forgetting directly in terms of the task distribution. Building on this identity, we establish an exponential decay guarantee for expected historical forgetting under every fixed task distribution in finite dimensions, characterize its asymptotic behavior, and relate decay to the distribution's coverage of observable directions. For an individual learned task, we show that subsequent tasks can collectively support recovery without exact revisits. We derive a lower bound on recovery time and construct a task distribution attaining its inverse-coverage scaling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

35 of 46 references.

References · 35

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Learning without Forgettingcited by 2,496
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  5. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  6. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  7. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  9. 2017Variational Continual Learningcited by 339
  10. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  11. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  12. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
23 more
  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
  2. 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 56
  3. 2023Continual Learning in Linear Classification on Separable Datacited by 34
  4. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
  5. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 30
  6. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
  7. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  8. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  9. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  10. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15
  11. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  12. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  13. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 12
  14. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  15. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  16. 2020Randomized Kaczmarz converges along small singular vectorscited by 9
  17. 2025Convergence and Implicit Bias of Gradient Descent on Continual Linear Classificationcited by 9
  18. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  19. 2025Replay Can Provably Increase Forgettingcited by 5
  20. 2025Analysis of Overparameterization in Continual Learning under a Linear Modelcited by 4
  21. 2025Are Greedy Task Orderings Better Than Random in Continual Linear Regression?cited by 3
  22. 2026Optimal L2 Regularization in High-dimensional Continual Linear Regressioncited by 3
  23. 2017Preasymptotic Convergence of Randomized Kaczmarz Methodcited by 1

Built on by

Not cited yet.