mlsanityData to 14 Oct 2026

Remember and Forget for Experience Replay

arXiv 1807.05827 ↗First posted 16 Jul 2018cs.LG stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors and can undermine the performance of ER. Many algorithms mitigate this issue by tuning hyper-parameters to slow down policy changes. An alternative is to actively enforce the similarity between policy and the experiences in the replay memory. We introduce Remember and Forget Experience Replay (ReF-ER), a novel method that can enhance RL algorithms with parameterized policies. ReF-ER (1) skips gradients computed from experiences that are too unlikely with the current policy and (2) regulates policy changes within a trust region of the replayed behaviors. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods. We find that ReF-ER consistently improves the performance of continuous-action, off-policy RL on fully observable benchmarks and partially observable flow control problems.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 11; 2023: 4; 2024: 8; 2025: 4; 2026: 6.

43 papers.

Sort:

By area:

  1. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  2. 2021Scientific multi-agent reinforcement learning for wall-models of turbulent flowscited by 52
  3. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 38
  4. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  5. 2021Learning Efficient Navigation in Vortical Flow Fieldscited by 26
  6. 2022Actor Prioritized Experience Replaycited by 20
  7. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  8. 2021Deep Reinforcement Learning with Quantum-inspired Experience Replaycited by 19
  9. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  10. 2023Curious Replay for Model-based Adaptationcited by 16
  11. 2024Prioritized Generative Replaycited by 13
  12. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
31 more
  1. 2021Simplifying Deep Reinforcement Learning via Self-Supervisioncited by 9
  2. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  3. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 7
  4. 2022Comparative analysis of machine learning methods for active flow controlcited by 6
  5. 2023Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learningcited by 6
  6. 2024Data Assimilation in Chaotic Systems Using Deep Reinforcement Learningcited by 5
  7. 2021Learning swimming escape patterns for larval fish under energy constraintscited by 3
  8. 2024Revisiting Experience Replayable Conditionscited by 3
  9. 2024ROER: Regularized Optimal Experience Replaycited by 3
  10. 2025Sample Efficient Experience Replay in Non-stationary Environmentscited by 3
  11. 2021Explaining Off-Policy Actor-Critic From A Bias-Variance Perspectivecited by 2
  12. 2022Remember and Forget Experience Replay for Multi-Agent Reinforcement Learningcited by 2
  13. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  14. 2022Multi-agent reinforcement learning for wall modeling in LES of flow over periodic hillscited by 2
  15. 2023Discovering Individual Rewards in Collective Behavior through Inverse Multi-Agent Reinforcement Learningcited by 2
  16. 2022Backpropagation through Time and Space: Learning Numerical Methods with Multi-Agent Reinforcement Learningcited by 1
  17. 2022Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reusecited by 1
  18. 2022Off-Policy Reinforcement Learning with Loss Function Weighted by Temporal Difference Errorcited by 1
  19. 2024Optimal navigation of magnetic artificial microswimmers in blood capillaries with deep reinforcement learningcited by 1
  20. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  21. 2025Gaze on the Prize: Shaping Visual Attention with Return-Guided Contrastive Learningcited by 1
  22. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  23. 2021Improving Experience Replay through Modeling of Similar Transitions' Setscited by 0
  24. 2022Game-theoretic Objective Space Planningcited by 0
  25. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiencescited by 0
  26. 2025Controlling Topological Defects in Polar Fluids via Reinforcement Learningcited by 0
  27. 2026VLM-Guided Experience Replaycited by 0
  28. 2026Variance Reduction Based Experience Replay for Policy Optimizationcited by 0
  29. 2026Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updatescited by 0
  30. 2026Cognitively-Grounded On-Device Runtime Learning for Ground Robots in Unknown Physical Environmentscited by 0
  31. 2026Uncertainty-Driven Replay Memory for Reinforcement Learningcited by 0