# Selective Experience Replay for Lifelong Learning

arXiv: https://arxiv.org/abs/1802.10269 · HTML: https://mlsanity.com/paper/1802.10269/

## Key facts

- Cited by 241 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2018.

## Built on by (most cited first, 30 of 241)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters](https://mlsanity.com/paper/2403.11549/) (2024)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation](https://mlsanity.com/paper/2208.05117/) (2022)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2303.06628/) (2023)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery](https://mlsanity.com/paper/2311.02058/) (2023)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [Don't Stop Learning: Towards Continual Learning for the CLIP Model](https://mlsanity.com/paper/2207.09248/) (2022)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Real-Time Evaluation in Online Continual Learning: A New Hope](https://mlsanity.com/paper/2302.01047/) (2023)
- [Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration](https://mlsanity.com/paper/2312.05229/) (2023)
- [SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection](https://mlsanity.com/paper/2402.17323/) (2024)
- [Same State, Different Task: Continual Reinforcement Learning without Interference](https://mlsanity.com/paper/2106.02940/) (2021)
- [Information-theoretic Online Memory Selection for Continual Learning](https://mlsanity.com/paper/2204.04763/) (2022)
- [Traj-MAE: Masked Autoencoders for Trajectory Prediction](https://mlsanity.com/paper/2303.06697/) (2023)
- [Modular Lifelong Reinforcement Learning via Neural Composition](https://mlsanity.com/paper/2207.00429/) (2022)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Continual Semantic Segmentation with Automatic Memory Sample Selection](https://mlsanity.com/paper/2304.05015/) (2023)
- [CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents](https://mlsanity.com/paper/2110.10067/) (2021)
- [Scaling Laws for Forgetting When Fine-Tuning Large Language Models](https://mlsanity.com/paper/2401.05605/) (2024)
- [SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models](https://mlsanity.com/paper/2401.08295/) (2024)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)
- [Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector](https://mlsanity.com/paper/2110.07905/) (2021)
- [Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2407.05342/) (2024)
- [Continuous Coordination As a Realistic Scenario for Lifelong Learning](https://mlsanity.com/paper/2103.03216/) (2021)

## Abstract (arXiv)

Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an experience replay process that augments the standard FIFO buffer and selectively stores experiences in a long-term memory. We explore four strategies for selecting which experiences will be stored: favoring surprise, favoring reward, matching the global training distribution, and maximizing coverage of the state space. We show that distribution matching successfully prevents catastrophic forgetting, and is consistently the best approach on all domains tested. While distribution matching has better and more consistent performance, we identify one case in which coverage maximization is beneficial - when tasks that receive less trained are more important. Overall, our results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting.