mlsanityData to 14 Oct 2026

Predicting the Susceptibility of Examples to Catastrophic Forgetting

arXiv 2406.09935 ↗First posted 14 Jun 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Catastrophic forgetting - the tendency of neural networks to forget previously learned data when learning new information - remains a central challenge in continual learning. In this work, we adopt a behavioral approach, observing a connection between learning speed and forgetting: examples learned more quickly are less prone to forgetting. Focusing on replay-based continual learning, we show that the composition of the replay buffer - specifically, whether it contains quickly or slowly learned examples - has a significant effect on forgetting. Motivated by this insight, we introduce Speed-Based Sampling (SBS), a simple yet general strategy that selects replay examples based on their learning speed. SBS integrates easily into existing buffer-based methods and improves performance across a wide range of competitive continual learning benchmarks, advancing state-of-the-art results. Our findings underscore the value of accounting for the forgetting dynamics when designing continual learning algorithms.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 52 of its 68 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

52 of 68 references.

References · 52

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  5. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2017Deep Image Priorcited by 1,098
  8. 2018Experience Replay for Continual Learningcited by 1,095
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  10. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  12. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
38 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  2. 2019Gradient based sample selection for online continual learningcited by 571
  3. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  4. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  5. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  6. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  7. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  10. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  11. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  13. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  14. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  15. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  16. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  17. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 133
  18. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  19. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  20. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  21. 2020Rethinking Experience Replay: a Bag of Tricks for Continual Learningcited by 94
  22. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  23. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 68
  24. 2022Active Learning Through a Covering Lenscited by 66
  25. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  26. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  27. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 56
  28. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 54
  29. 2019Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generatorscited by 40
  30. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  31. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  32. 2019A comprehensive, application-oriented study of catastrophic forgetting in DNNscited by 35
  33. 2020Class-incremental Learning with Pre-allocated Fixed Classifierscited by 33
  34. 2023How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budgetcited by 17
  35. 2020Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networkscited by 15
  36. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  37. 2018Iterative Projection and Matching: Finding Structure-preserving Representatives and Its Application to Computer Visioncited by 6
  38. 2020Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualizationcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Same accuracy, twice as fast: continuous training surpasses retraining from scratchcited by 4
  2. 2025What is the role of memorization in Continual Learning?cited by 0
  3. 2025ZeBROD: Zero-Retraining Based Recognition and Object Detection Frameworkcited by 0
  4. 2026Memory-efficient Continual Learning with Prototypical Exemplar Condensationcited by 0
  5. 2026X-Edit: Exact, Explicit, and Explainable Null-Space Editing for Medical Vision Transformerscited by 0
  6. 2026Geometry-Anchored Transport Framework for Exemplar-Free Class-Incremental Learningcited by 0
  7. 2026Age of Learning: Temporal Persistence of Prediction Errors as a Learning Signalcited by 0