# Reliability-Adjusted Prioritized Experience Replay

arXiv: https://arxiv.org/abs/2506.18482 · HTML: https://mlsanity.com/paper/2506.18482/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 28 of 28)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replay](https://mlsanity.com/paper/1903.04566/) (2019)
- [Experience Replay Optimization](https://mlsanity.com/paper/1906.08387/) (2019)
- [Atari-5: Distilling the Arcade Learning Environment down to Five Games](https://mlsanity.com/paper/2210.02019/) (2022)
- [EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data](https://mlsanity.com/paper/2403.00564/) (2024)
- [Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update](https://mlsanity.com/paper/1805.12375/) (2018)
- [Prioritized Sequence Experience Replay](https://mlsanity.com/paper/1905.12726/) (2019)
- [Learning to Sample with Local and Global Contexts in Experience Replay Buffer](https://mlsanity.com/paper/2007.07358/) (2020)
- [Uncertainty Prioritized Experience Replay](https://mlsanity.com/paper/2506.09270/) (2025)
- [Investigating the Interplay of Prioritized Replay and Generalization](https://mlsanity.com/paper/2407.09702/) (2024)

## Built on by (most cited first, 3 of 3)

- [Replay-buffer engineering for noise-aware quantum circuit optimization](https://mlsanity.com/paper/2604.21863/) (2026)
- [Target-Aligned Reinforcement Learning](https://mlsanity.com/paper/2603.29501/) (2026)
- [When Does Non-Uniform Replay Matter in Reinforcement Learning?](https://mlsanity.com/paper/2605.10236/) (2026)

## Abstract (arXiv)

Experience replay enables data-efficient learning from past experiences in online reinforcement learning agents. Traditionally, experiences were sampled uniformly from a replay buffer, regardless of differences in experience-specific learning potential. In an effort to sample more efficiently, researchers introduced Prioritized Experience Replay (PER). In this paper, we propose an extension to PER by introducing a novel measure of temporal difference error reliability. We theoretically show that the resulting transition selection algorithm, Reliability-adjusted Prioritized Experience Replay (ReaPER), enables more efficient learning than PER. We further present empirical results showing that ReaPER outperforms PER across various environment types, including the Atari-10 benchmark.