mlsanityData to 7 Oct 2026

Revisiting Fundamentals of Experience Replay

arXiv 2007.06700 ↗First posted 13 Jul 2020cs.LG stat.MLcited by 171

Not on the map. Why?

Abstract (arXiv)

Experience replay is central to off-policy algorithms in deep reinforcement learning (RL), but there remain significant gaps in our understanding. We therefore present a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (replay ratio). Our additive and ablative studies upend conventional wisdom around experience replay -- greater capacity is found to substantially increase the performance of certain algorithms, while leaving others unaffected. Counterintuitively we show that theoretically ungrounded, uncorrected n-step returns are uniquely beneficial while other techniques confer limited benefit for sifting through larger memory. Separately, by directly controlling the replay ratio we contextualize previous observations in the literature and empirically measure its importance across a variety of deep RL algorithms. Finally, we conclude by testing a set of hypotheses on the nature of these performance benefits.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 24; 2023: 33; 2024: 28; 2025: 31; 2026: 26.

171 papers.

Sort:

By area:

  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  3. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  4. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  5. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  6. 2023Synthetic Experience Replaycited by 100
  7. 2025Reinforcement Learning with Action Chunkingcited by 97
  8. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  9. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  10. 2023Towards Understanding and Improving GFlowNet Trainingcited by 74
  11. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  12. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
38 more
  1. 2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 65
  2. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  3. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
  4. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  5. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  6. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  7. 2023Small batch deep reinforcement learningcited by 35
  8. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  9. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  10. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  11. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  12. 2021Reverb: A Framework For Experience Replaycited by 27
  13. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  14. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  15. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  16. 2022Topological Experience Replaycited by 17
  17. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  18. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  19. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  20. 2021Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillationcited by 15
  21. 2021Large Batch Experience Replaycited by 14
  22. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  23. 2023Order-Preserving GFlowNetscited by 13
  24. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  25. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  26. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  27. 2021Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscited by 11
  28. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  29. 2021Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learningcited by 10
  30. 2024RetroGFN: Diverse and Feasible Retrosynthesis using GFlowNetscited by 10
  31. 2021Solving Heterogeneous General Equilibrium Economic Models with Deep Reinforcement Learningcited by 9
  32. 2022Analysis of Stochastic Processes through Replay Bufferscited by 9
  33. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  34. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  35. 2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8
  36. 2022A Rubric for Human-like Agents and NeuroAIcited by 8
  37. 2023Maximum diffusion reinforcement learningcited by 8
  38. 2023Hieros: Hierarchical Imagination on Structured State Space Sequence World Modelscited by 8

50 most cited of 171 All 171 (JSON)