Towards guarantees for parameter isolation in continual learning
Not on the map. Why?
Abstract (arXiv)
Deep learning has proved to be a successful paradigm for solving many challenges in machine learning. However, deep neural networks fail when trained sequentially on multiple tasks, a shortcoming known as catastrophic forgetting in the continual learning literature. Despite a recent flourish of learning algorithms successfully addressing this problem, we find that provable guarantees against catastrophic forgetting are lacking. In this work, we study the relationship between learning and forgetting by looking at the geometry of neural networks' loss landscape. We offer a unifying perspective on a family of continual learning algorithms, namely methods based on parameter isolation, and we establish guarantees on catastrophic forgetting for some of them.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 2 Oct 2023.
Builds on
36 of 48 references.
References · 36
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2016Progressive Neural Networkscited by 1,361
- 2017Continual Learning with Deep Generative Replaycited by 1,158
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
- 2019Orthogonal Gradient Descent for Continual Learningcited by 369
- 2021Gradient Projection Memory for Continual Learningcited by 348
24 more
- 2017Variational Continual Learningcited by 339
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
- 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
- 2020Supermasks in Superpositioncited by 227
- 2018Gradient Descent Happens in a Tiny Subspacecited by 217
- 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
- 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
- 2018SpaceNet: A Remote Sensing Dataset and Challenge Seriescited by 179
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
- 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
- 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 102
- 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
- 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
- 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 72
- 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
- 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
- 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
- 2016Gradient Descent Converges to Minimizerscited by 25
- 2017Accelerated Stochastic Power Iterationcited by 22
- 2022Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activationscited by 19
- 2018Large batch size training of neural networks with adversarial training and second-order informationcited by 18
- 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
Built on by
Not cited yet.