mlsanityData to 14 Oct 2026

Learn the Time to Learn: Replay Scheduling in Continual Learning

arXiv 2209.08660 ↗First posted 18 Sep 2022cs.LG cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Replay methods are known to be successful at mitigating catastrophic forgetting in continual learning scenarios despite having limited access to historical data. However, storing historical data is cheap in many real-world settings, yet replaying all historical data is often prohibited due to processing time constraints. In such settings, we propose that continual learning systems should learn the time to learn and schedule which tasks to replay at different time steps. We first demonstrate the benefits of our proposal by using Monte Carlo tree search to find a proper replay schedule, and show that the found replay schedules can outperform fixed scheduling policies when combined with various replay methods in different continual learning settings. Additionally, we propose a framework for learning replay scheduling policies with reinforcement learning. We show that the learned policies can generalize better in new continual learning scenarios compared to equally replaying all seen tasks, without added computational cost. Our study reveals the importance of learning the time to learn in continual learning, which brings current research closer to real-world needs.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 74 of its 112 references that are in the map. First posted on arXiv 18 Sep 2022.

Builds on

74 of 112 references.

References · 74

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2016Learning without Forgettingcited by 2,496
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  11. 2016Matching Networks for One Shot Learningcited by 2,255
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
38 more
  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  2. 2016Progressive Neural Networkscited by 1,361
  3. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  4. 2017Continual Learning with Deep Generative Replaycited by 1,158
  5. 2018Experience Replay for Continual Learningcited by 1,095
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  7. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  8. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  11. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  12. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  13. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  14. 2019Three scenarios for continual learningcited by 614
  15. 2019Gradient based sample selection for online continual learningcited by 571
  16. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  17. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  18. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
  19. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  20. 2017Variational Continual Learningcited by 339
  21. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  22. 2019Continual learning with hypernetworkscited by 273
  23. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  24. 2018Selective Experience Replay for Lifelong Learningcited by 241
  25. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  26. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  27. 2018Reinforced Continual Learningcited by 186
  28. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  29. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 161
  30. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  31. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  32. 2020Adversarial Continual Learningcited by 142
  33. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  34. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  35. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  36. 2018Generalization and Regularization in DQNcited by 118
  37. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
  38. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 0; 2026: 6.

11 papers.

  1. 2024Continual Learning and Catastrophic Forgettingcited by 77
  2. 2026MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuningcited by 4
  3. 2023Adiabatic replay for continual learningcited by 3
  4. 2026TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFTcited by 1
  5. 2026RAIE: Region-Aware Incremental Preference Editing with LoRA for LLM-based Recommendationcited by 1
  6. 2026Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptationcited by 1
  7. 2024Brain-Inspired Continual Learning-Robust Feature Distillation and Re-Consolidation for Class Incremental Learningcited by 0
  8. 2024Adaptive Variational Continual Learning via Task-Heuristic Modellingcited by 0
  9. 2024Low-Rank Filtering and Smoothing for Sequential Deep Learningcited by 0
  10. 2026Beyond Class Marginals: Bounding Rehearsal Gaps without Freezing Class Co-occurrencecited by 0
  11. 2026Replay in the Silent Degrees of Freedom: Continual Learning Without an Offline Phasecited by 0