mlsanityData to 14 Oct 2026

A Tighter Convergence Proof of Reverse Experience Replay

arXiv 2408.16999 ↗First posted 30 Aug 2024cs.LG stat.MLcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In reinforcement learning, Reverse Experience Replay (RER) is a recently proposed algorithm that attains better sample complexity than the classic experience replay method. RER requires the learning algorithm to update the parameters through consecutive state-action-reward tuples in reverse order. However, the most recent theoretical analysis only holds for a minimal learning rate and short consecutive steps, which converge slower than those large learning rate algorithms without RER. In view of this theoretical and empirical gap, we provide a tighter analysis that mitigates the limitation on the learning rate and the length of consecutive steps. Furthermore, we show theoretically that RER converges with a larger learning rate and a longer sequence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 29 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

17 of 29 references.

References · 17

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  5. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 181
  6. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 95
  7. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  8. 2020Least Squares Regression with Markovian Data: Fundamental Limits and Algorithmscited by 56
  9. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  10. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  11. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  12. 2022Actor Prioritized Experience Replaycited by 20
5 more
  1. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19
  2. 2021Streaming Linear System Identification with Reverse Experience Replaycited by 17
  3. 2021Large Batch Experience Replaycited by 14
  4. 2014Rate of Convergence and Error Bounds for LSTD(λ)cited by 8
  5. 2019Reverse Experience Replaycited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Variance Reduction via Resampling and Experience Replay
  2. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms
  3. 2025Experience Replay with Random Reshuffling
  4. 2025Reliability-Adjusted Prioritized Experience Replay
  5. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?
  6. 2024ROER: Regularized Optimal Experience Replay
  7. 2024Revisiting Experience Replayable Conditions
  8. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learning