# Remember and Forget for Experience Replay

arXiv: https://arxiv.org/abs/1807.05827 · HTML: https://mlsanity.com/paper/1807.05827/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2018.

## Built on by (most cited first, 30 of 43)

- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [Scientific multi-agent reinforcement learning for wall-models of turbulent flows](https://mlsanity.com/paper/2106.11144/) (2021)
- [Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies](https://mlsanity.com/paper/2111.02552/) (2021)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Learning Efficient Navigation in Vortical Flow Fields](https://mlsanity.com/paper/2102.10536/) (2021)
- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization](https://mlsanity.com/paper/2302.10418/) (2023)
- [Deep Reinforcement Learning with Quantum-inspired Experience Replay](https://mlsanity.com/paper/2101.02034/) (2021)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Prioritized Generative Replay](https://mlsanity.com/paper/2410.18082/) (2024)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Simplifying Deep Reinforcement Learning via Self-Supervision](https://mlsanity.com/paper/2106.05526/) (2021)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [Comparative analysis of machine learning methods for active flow control](https://mlsanity.com/paper/2202.11664/) (2022)
- [Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning](https://mlsanity.com/paper/2305.02054/) (2023)
- [Data Assimilation in Chaotic Systems Using Deep Reinforcement Learning](https://mlsanity.com/paper/2401.00916/) (2024)
- [Learning swimming escape patterns for larval fish under energy constraints](https://mlsanity.com/paper/2105.00771/) (2021)
- [Revisiting Experience Replayable Conditions](https://mlsanity.com/paper/2402.10374/) (2024)
- [ROER: Regularized Optimal Experience Replay](https://mlsanity.com/paper/2407.03995/) (2024)
- [Sample Efficient Experience Replay in Non-stationary Environments](https://mlsanity.com/paper/2509.15032/) (2025)
- [Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective](https://mlsanity.com/paper/2110.02421/) (2021)
- [Remember and Forget Experience Replay for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.13319/) (2022)
- [On the Reuse Bias in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2209.07074/) (2022)
- [Multi-agent reinforcement learning for wall modeling in LES of flow over periodic hills](https://mlsanity.com/paper/2211.16427/) (2022)
- [Discovering Individual Rewards in Collective Behavior through Inverse Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.10548/) (2023)
- [Backpropagation through Time and Space: Learning Numerical Methods with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.08937/) (2022)
- [Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reuse](https://mlsanity.com/paper/2206.13714/) (2022)
- [Off-Policy Reinforcement Learning with Loss Function Weighted by Temporal Difference Error](https://mlsanity.com/paper/2212.13175/) (2022)

## Abstract (arXiv)

Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors and can undermine the performance of ER. Many algorithms mitigate this issue by tuning hyper-parameters to slow down policy changes. An alternative is to actively enforce the similarity between policy and the experiences in the replay memory. We introduce Remember and Forget Experience Replay (ReF-ER), a novel method that can enhance RL algorithms with parameterized policies. ReF-ER (1) skips gradients computed from experiences that are too unlikely with the current policy and (2) regulates policy changes within a trust region of the replayed behaviors. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods. We find that ReF-ER consistently improves the performance of continuous-action, off-policy RL on fully observable benchmarks and partially observable flow control problems.