mlsanityData to 14 Oct 2026

Reliability-Adjusted Prioritized Experience Replay

arXiv 2506.18482 ↗First posted 23 Jun 2025cs.LG stat.MLcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Experience replay enables data-efficient learning from past experiences in online reinforcement learning agents. Traditionally, experiences were sampled uniformly from a replay buffer, regardless of differences in experience-specific learning potential. In an effort to sample more efficiently, researchers introduced Prioritized Experience Replay (PER). In this paper, we propose an extension to PER by introducing a novel measure of temporal difference error reliability. We theoretically show that the resulting transition selection algorithm, Reliability-adjusted Prioritized Experience Replay (ReaPER), enables more efficient learning than PER. We further present empirical results showing that ReaPER outperforms PER across various environment types, including the Atari-10 benchmark.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 43 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

28 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Investigating the Interplay of Prioritized Replay and Generalization12 Jul
  2. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data1 Mar

Other references · 26

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2023Mastering Diverse Domains through World Modelscited by 1,253
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2018Experience Replay for Continual Learningcited by 1,095
  8. 2016An overview of gradient descent optimization algorithmscited by 1,033
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  12. 2018Distributed Prioritized Experience Replaycited by 269
14 more
  1. 2018Selective Experience Replay for Lifelong Learningcited by 241
  2. 2021Mastering Atari Games with Limited Datacited by 237
  3. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  4. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  5. 2020Revisiting Fundamentals of Experience Replaycited by 175
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2017A Deeper Look at Experience Replaycited by 105
  8. 2019Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replaycited by 51
  9. 2019Experience Replay Optimizationcited by 48
  10. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  11. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  12. 2019Prioritized Sequence Experience Replaycited by 20
  13. 2020Learning to Sample with Local and Global Contexts in Experience Replay Buffercited by 10
  14. 2025Uncertainty Prioritized Experience Replaycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Replay-buffer engineering for noise-aware quantum circuit optimizationcited by 1
  2. 2026Target-Aligned Reinforcement Learningcited by 0
  3. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
  2. 2024ROER: Regularized Optimal Experience Replay
  3. 2025Variance Reduction via Resampling and Experience Replay
  4. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  5. 2024Prioritized Generative Replay
  6. 2025Sample Efficient Experience Replay in Non-stationary Environments
  7. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learning
  8. 2024A Tighter Convergence Proof of Reverse Experience Replay