mlsanityData to 7 Oct 2026

Data-dependent and Oracle Bounds on Forgetting in Continual Learning

arXiv 2406.09370 ↗First posted 13 Jun 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In continual learning, knowledge must be preserved and re-used between tasks, maintaining good transfer to future tasks and minimizing forgetting of previously learned ones. While several practical algorithms have been devised for this setting, there have been few theoretical works aiming to quantify and bound the degree of Forgetting in general settings. For exemplar-free methods, we provide both data-dependent upper bounds that apply regardless of model and algorithm choice, and oracle bounds for Gibbs posteriors. We derive an algorithm based on our bounds and demonstrate empirically that our approach yields tight and practical bounds on forgetting for several continual learning problems and algorithms.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

25 of 39 references.

References · 25

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2016Learning without Forgettingcited by 2,465
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  6. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  8. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  9. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  10. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  11. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  12. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
13 more
  1. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  2. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  3. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  4. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  5. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  6. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  7. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 35
  8. 2023A unified recipe for deriving (time-uniform) PAC-Bayes boundscited by 25
  9. 2022PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingalescited by 23
  10. 2022Online PAC-Bayes Learningcited by 22
  11. 2022A Theory for Knowledge Transfer in Continual Learningcited by 16
  12. 2020Beyond H-Divergence: Domain Adaptation Theory With Jensen-Shannon Divergencecited by 15
  13. 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Self-Evolving Agents with Anytime-Valid Certificatescited by 6
  2. 2025Sample Compression for Self Certified Continual Learningcited by 1
  3. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 0