mlsanityData to 7 Oct 2026

Continual Learning in Linear Classification on Separable Data

arXiv 2306.03534 ↗First posted 6 Jun 2023cs.LG cs.NA math.NAcited by 31

Not on the map. Why?

Abstract (arXiv)

We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

32 of 61 references.

References · 32

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2021Convex optimizationcited by 2,479
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2018Online Learning: A Comprehensive Surveycited by 203
  11. 2018To understand deep learning we need to understand kernel learningcited by 186
  12. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
20 more
  1. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  2. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  3. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  4. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  5. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  6. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  7. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  8. 2014Curriculum Learning of Multiple Taskscited by 65
  9. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  10. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  11. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  12. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  13. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  14. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  15. 2021Is Class-Incremental Enough for Continual Learning?cited by 22
  16. 2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19
  17. 2023Is forgetting less a good inductive bias for forward transfer?cited by 18
  18. 2020Sampling Kaczmarz Motzkin Method for Linear Feasibility Problems: Generalization & Accelerationcited by 5
  19. 2013Analysis of the convergence rate for the cyclic projection algorithm applied to basic semi-algebraic convex setscited by 1
  20. 2022Polynomial Estimates for the Method of Cyclic Projections in Hilbert Spacescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 14; 2026: 6.

31 papers.

Sort:

  1. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  2. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  3. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  4. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  5. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  6. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  7. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
  8. 2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 10
  9. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  10. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 8
  11. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  12. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
19 more
  1. 2025Analysis of Overparameterization in Continual Learning under a Linear Modelcited by 4
  2. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 4
  3. 2025Replay Can Provably Increase Forgettingcited by 4
  4. 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
  5. 2025A statistical physics framework for optimal learningcited by 3
  6. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 3
  7. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 2
  8. 2025Data Heterogeneity and Forgotten Labels in Split Federated Learningcited by 2
  9. 2024Statistical curriculum learning: An elimination algorithm achieving an oracle riskcited by 1
  10. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  11. 2025On Understanding of the Dynamics of Model Capacity in Continual Learningcited by 1
  12. 2026Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learningcited by 1
  13. 2024Adaptive Margin Global Classifier for Exemplar-Free Class-Incremental Learningcited by 0
  14. 2024Towards Macro-AUC oriented Imbalanced Multi-Label Continual Learningcited by 0
  15. 2025The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learningcited by 0
  16. 2026Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacitiescited by 0
  17. 2026From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learningcited by 0
  18. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  19. 2026The Sequential Price of Continual Learningcited by 0