mlsanityData to 14 Oct 2026

Convergence of Continual Learning in Homogeneous Deep Networks

arXiv 2606.30559 ↗First posted 29 Jun 2026cs.LG cs.NA math.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

43 of 63 references.

References · 43

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  4. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  9. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  10. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  11. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  12. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
31 more
  1. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  2. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
  3. 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
  4. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  5. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  6. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  7. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  8. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  9. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
  10. 2023Continual Learning in Linear Classification on Separable Datacited by 34
  11. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
  12. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
  13. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  14. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  15. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  16. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  17. 2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19
  18. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
  19. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 14
  20. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  21. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  22. 2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 12
  23. 2022Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructionscited by 11
  24. 2024Continual Learning in the Presence of Repetitioncited by 10
  25. 2025Convergence and Implicit Bias of Gradient Descent on Continual Linear Classificationcited by 9
  26. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  27. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  28. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 3
  29. 2025Are Greedy Task Orderings Better Than Random in Continual Linear Regression?cited by 3
  30. 2026Optimal L2 Regularization in High-dimensional Continual Linear Regressioncited by 3
  31. 2016Linear Convergence of Projection Algorithmscited by 1

Built on by

Not cited yet.