mlsanityData to 7 Oct 2026

Experience Replay for Continual Learning

arXiv 1811.11682 ↗First posted 28 Nov 2018cs.LG cs.AI stat.MLcited by 1,074

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

Key facts. Cited by 1,074 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 74; 2022: 124; 2023: 170; 2024: 223; 2025: 221; 2026: 262.

1,074 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  4. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  5. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  6. 2022Memory-Based Model Editing at Scalecited by 443
  7. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  8. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  11. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  12. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
38 more
  1. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  2. 2022Patching open-vocabulary models by interpolating weightscited by 187
  3. 2023Mitigating the Alignment Tax of RLHFcited by 181
  4. 2023Understanding plasticity in neural networkscited by 170
  5. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  6. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  7. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  8. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 130
  9. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  10. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  11. 2022Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptationcited by 98
  12. 2023A Definition of Continual Reinforcement Learningcited by 98
  13. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  14. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  15. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  16. 2022A Theoretical Study on Solving Continual Learningcited by 78
  17. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  18. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  19. 2024Continual Learning and Catastrophic Forgettingcited by 73
  20. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  21. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  22. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  23. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
  24. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  25. 2023IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturingcited by 64
  26. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  27. 2021Continual Learning via Local Module Compositioncited by 61
  28. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  29. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  30. 2023Continual Learning: Applications and the Road Forwardcited by 59
  31. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  32. 2023Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learningcited by 55
  33. 2021Subspace Regularizers for Few-Shot Class Incremental Learningcited by 54
  34. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  35. 2022An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditionscited by 54
  36. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  37. 2023Better Generative Replay for Continual Federated Learningcited by 53
  38. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53

50 most cited of 1,074 All 1,074 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
  2. 2019Continual Learning Using World Models for Pseudo-Rehearsal
  3. 2026Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
  4. 2025Mitigating Catastrophic Forgetting and Mode Collapse in Text-to-Image Diffusion via Latent Replay
  5. 2026Rethinking Transfer in Continual Learning: A Replay-Based Realisation
  6. 2026When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents
  7. 2024Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay
  8. 2024Joint Diffusion models in Continual Learning