Data to 14 Oct 2026

A Deeper Look at Experience Replay

arXiv 1712.01275 ↗First posted 4 Dec 2017cs.LG cs.AIcited by 105

Not on the map. Why?

Cited by 25 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, in this paper we rethink the utility of experience replay. It introduces a new hyper-parameter, the memory buffer size, which needs carefully tuning. However unfortunately the importance of this new hyper-parameter has been underestimated in the community for a long time. In this paper we did a systematic empirical study of experience replay under various function representations. We showcase that a large replay buffer can significantly hurt the performance. Moreover, we propose a simple O(1) method to remedy the negative influence of a large replay buffer. We showcase its utility in both simple grid world and challenging domains like Atari games.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 25; 2022: 16; 2023: 19; 2024: 14; 2025: 13; 2026: 16.

105 papers.

Sort:

By area:

2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 66
2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 56
2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 35
2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
2021Continual Learning on Noisy Data Streams via Self-Purified Replaycited by 27
2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
2022Actor Prioritized Experience Replaycited by 20
2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20

38 more

2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
2021Large Batch Experience Replaycited by 14
2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
2022Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2cited by 11
2022Beyond Supervised Continual Learning: a Reviewcited by 11
2025ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulationcited by 11
2021Simplifying Deep Reinforcement Learning via Self-Supervisioncited by 9
2022Analysis of Stochastic Processes through Replay Bufferscited by 9
2023Maximum diffusion reinforcement learningcited by 9
2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
2026Efficient RL Training for LLMs with Experience Replaycited by 9
2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8
2023Fair yet Asymptotically Equal Collaborative Learningcited by 8
2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
2021Sim and Real: Better Togethercited by 7
2022DouZero+: Improving DouDizhu AI by Opponent Modeling and Coach-guided Learningcited by 7
2022Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problemscited by 6
2019ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrationscited by 5
2021Self-Improving Semantic Perception for Indoor Localisationcited by 5
2021Convergence Results For Q-Learning With Experience Replaycited by 5
2022Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learningcited by 5
2022Learning from Data Streams: An Overview and Updatecited by 5
2024Scalable Reinforcement Learning-based Neural Architecture Searchcited by 5
2022Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environmentscited by 4
2022Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learningcited by 4
2023DQNAS: Neural Architecture Search using Reinforcement Learningcited by 4
2021Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learningcited by 3
2021Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learningcited by 3
2021Modeling Interactions of Autonomous Vehicles and Pedestrians with Deep Multi-Agent Reinforcement Learning for Collision Avoidancecited by 3
2021Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replaycited by 3
2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
2022System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Gamescited by 3
2023A Survey of Historical Learning: Learning Models with Learning Historycited by 3
2024ROER: Regularized Optimal Experience Replaycited by 3
2024SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planningcited by 3
2025Reliability-Adjusted Prioritized Experience Replaycited by 3

50 most cited of 105 All 105 (JSON)