# Reverse Experience Replay

arXiv: https://arxiv.org/abs/1910.08780 · HTML: https://mlsanity.com/paper/1910.08780/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs](https://mlsanity.com/paper/2110.08440/) (2021)
- [Streaming Linear System Identification with Reverse Experience Replay](https://mlsanity.com/paper/2103.05896/) (2021)
- [Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systems](https://mlsanity.com/paper/2105.11558/) (2021)
- [Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation](https://mlsanity.com/paper/2405.18688/) (2024)
- [Introspective Experience Replay: Look Back When Surprised](https://mlsanity.com/paper/2206.03171/) (2022)
- [Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?](https://mlsanity.com/paper/2312.17168/) (2023)
- [Finite-Time Analysis of Temporal Difference Learning with Experience Replay](https://mlsanity.com/paper/2306.09746/) (2023)
- [A Tighter Convergence Proof of Reverse Experience Replay](https://mlsanity.com/paper/2408.16999/) (2024)

## Abstract (arXiv)

This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay memory capacity, Deep Q-learning Network with Reverse Experience Replay shows competitive results against both Double DQN, with a standard Experience Replay, and vanilla DQN. Also, RER achieves significantly increased results in tasks with a lack of experience and Replay memory capacity.