mlsanityData to 7 Oct 2026

Slowing Down Forgetting in Continual Learning

arXiv 2411.06916 ↗First posted 11 Nov 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

A common challenge in continual learning (CL) is catastrophic forgetting, where the performance on old tasks drops after new, additional tasks are learned. In this paper, we propose a novel framework called ReCL to slow down forgetting in CL. Our framework exploits an implicit bias of gradient-based neural networks due to which these converge to margin maximization points. Such convergence points allow us to reconstruct old data from previous tasks, which we then combine with the current training data. Our framework is flexible and can be applied on top of existing, state-of-the-art CL methods. We further demonstrate the performance gain from our framework across a large series of experiments, including two challenging CL scenarios (class incremental and domain incremental learning), different datasets (MNIST, CIFAR10, TinyImagenet), and different network architectures. Across all experiments, we find large performance gains through ReCL. To the best of our knowledge, our framework is the first to address catastrophic forgetting by leveraging models in CL as their own memory buffers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 60 of its 94 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

60 of 94 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2016Learning without Forgettingcited by 2,465
  8. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
38 more
  1. 2017Continual Learning with Deep Generative Replaycited by 1,152
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  5. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  8. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  9. 2019Three scenarios for continual learningcited by 610
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2021Gradient Projection Memory for Continual Learningcited by 338
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  13. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  14. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  15. 2020Supermasks in Superpositioncited by 223
  16. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  17. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  18. 2020Directional convergence and alignment in deep learningcited by 172
  19. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  20. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  21. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  22. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  23. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  24. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  25. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  26. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  27. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  28. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  29. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  30. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  31. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  32. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  33. 2021ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learningcited by 68
  34. 2022Class-Incremental Learning with Strong Pre-trained Modelscited by 60
  35. 2021Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworkscited by 56
  36. 2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49
  37. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  38. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43

50 most cited of 60 All 60 (JSON)

Built on by

Not cited yet.