On the Theory of Continual Learning with Gradient Descent for Neural Networks
Not on the map. Why?
Abstract (arXiv)
Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting earlier ones, is a central goal of artificial intelligence. To better understand its underlying mechanisms, we study the limitations of continual learning in a tractable yet representative setting. Specifically, we analyze one-hidden-layer quadratic neural networks trained by gradient descent on a sequence of XOR-cluster datasets with Gaussian noise, where different tasks correspond to clusters with orthogonal means. Our analysis is based on a tight characterization of gradient descent dynamics for the training loss, which yields explicit bounds on the rate of train-time forgetting as functions of the number of iterations, sample size, number of tasks, and hidden-layer width. We then leverage an algorithmic stability framework to bound the generalization gap, leading to corresponding guarantees on test-time forgetting. Together, our results provide the first closed-form guarantees for forgetting in continual learning with neural networks and show how key problem parameters jointly govern forgetting dynamics. Numerical experiments corroborate our theoretical results.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 52 references that are in the map. First posted on arXiv 7 Oct 2025.
Builds on
45 of 52 references.
References · 45
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Learning without Forgettingcited by 2,465
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
- 2018Experience Replay for Continual Learningcited by 1,074
- 2015Prioritized Experience Replaycited by 1,070
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
- 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
- 2021Deep learning: a statistical viewpointcited by 239
- 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
33 more
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
- 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 118
- 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
- 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
- 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
- 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
- 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
- 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
- 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
- 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
- 2022Architecture Matters in Continual Learningcited by 46
- 2020On the training dynamics of deep networks with L2 regularizationcited by 38
- 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
- 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
- 2023Continual Learning in Linear Classification on Separable Datacited by 31
- 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
- 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 25
- 2019Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problemscited by 23
- 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
- 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
- 2021Provable Lifelong Learning of Representationscited by 15
- 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
- 2024On the Diminishing Returns of Width for Continual Learningcited by 12
- 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
- 2022Theoretical Understanding of the Information Flow on Continual Learning Performancecited by 8
- 2024Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methodscited by 7
- 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
- 2024Continual learning with the neural tangent ensemblecited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.