# The Effects of Memory Replay in Reinforcement Learning

arXiv: https://arxiv.org/abs/1710.06574 · HTML: https://mlsanity.com/paper/1710.06574/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2017.

## Built on by (most cited first, 23 of 23)

- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization](https://mlsanity.com/paper/2302.10418/) (2023)
- [Adaptive Informative Path Planning Using Deep Reinforcement Learning for UAV-based Active Sensing](https://mlsanity.com/paper/2109.13570/) (2021)
- [Analysis of Stochastic Processes through Replay Buffers](https://mlsanity.com/paper/2206.12848/) (2022)
- [ReMindRAG: Low-Cost LLM-Guided Knowledge Graph Traversal for Efficient RAG](https://mlsanity.com/paper/2510.13193/) (2025)
- [Sim and Real: Better Together](https://mlsanity.com/paper/2110.00445/) (2021)
- [Uncertainty Prioritized Experience Replay](https://mlsanity.com/paper/2506.09270/) (2025)
- [Convergence Results For Q-Learning With Experience Replay](https://mlsanity.com/paper/2112.04213/) (2021)
- [Q-Learning based system for path planning with unmanned aerial vehicles swarms in obstacle environments](https://mlsanity.com/paper/2303.17655/) (2023)
- [Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2102.11319/) (2021)
- [Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning](https://mlsanity.com/paper/2109.03975/) (2021)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Transformer with Memory Replay](https://mlsanity.com/paper/2205.09869/) (2022)
- [Timing Process Interventions with Causal Inference and Reinforcement Learning](https://mlsanity.com/paper/2306.04299/) (2023)
- [Attention Loss Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2309.06684/) (2023)
- [Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences](https://mlsanity.com/paper/2405.14629/) (2024)
- [An Efficient and Uncertainty-aware Reinforcement Learning Framework for Quality Assurance in Extrusion Additive Manufacturing](https://mlsanity.com/paper/2503.00971/) (2025)
- [Replay For Safety](https://mlsanity.com/paper/2112.04229/) (2021)
- [Automatic Discovery of Multi-perspective Process Model using Reinforcement Learning](https://mlsanity.com/paper/2211.16687/) (2022)
- [Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropout](https://mlsanity.com/paper/2301.11168/) (2023)
- [MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance](https://mlsanity.com/paper/2602.17930/) (2026)
- [Live LTL Progress Tracking: Towards Task-Based Exploration](https://mlsanity.com/paper/2604.17106/) (2026)
- [Efficient Long-Horizon Learning for Learned Optimization](https://mlsanity.com/paper/2607.06772/) (2026)

## Abstract (arXiv)

Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. How does the amount of memory kept affect learning dynamics? Does it help to prioritize certain experiences? In this paper, we address these questions by formulating a dynamical systems ODE model of Q-learning with experience replay. We derive analytic solutions of the ODE for a simple setting. We show that even in this very simple setting, the amount of memory kept can substantially affect the agent's performance. Too much or too little memory both slow down learning. Moreover, we characterize regimes where prioritized replay harms the agent's learning. We show that our analytic solutions have excellent agreement with experiments. Finally, we propose a simple algorithm for adaptively changing the memory buffer size which achieves consistently good empirical performance.