mlsanityData to 14 Oct 2026

Prioritized Experience Replay

arXiv 1511.05952 ↗First posted 18 Nov 2015cs.LGcited by 1,099

Not on the map. Why?

Cited by 247 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.

Key facts. Cited by 1,099 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 5; 2019: 2; 2020: 3; 2021: 220; 2022: 186; 2023: 170; 2024: 158; 2025: 177; 2026: 178.

1,099 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  4. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  6. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  7. 2022Transformers are Sample-Efficient World Modelscited by 328
  8. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  11. 2021Mastering Atari Games with Limited Datacited by 237
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
38 more
  1. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  2. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  3. 2023FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Leadcited by 175
  4. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  5. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 160
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  8. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  9. 2018Self-Imitation Learningcited by 138
  10. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  11. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  12. 2022Flow Annealed Importance Sampling Bootstrapcited by 127
  13. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  14. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  15. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  16. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  17. 2022Exploring Chemical Space with Score-based Out-of-distribution Generationcited by 95
  18. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  19. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  20. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  21. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  22. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  23. 2021Learning and Planning in Complex Action Spacescited by 77
  24. 2021Off-Belief Learningcited by 75
  25. 2023Towards Understanding and Improving GFlowNet Trainingcited by 75
  26. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  27. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  28. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  29. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  30. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  31. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  32. 2021Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generationcited by 68
  33. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  34. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  35. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  36. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  37. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 61
  38. 2023Continual Learning: Applications and the Road Forwardcited by 59

50 most cited of 1,099 All 1,099 (JSON)