mlsanityData to 14 Oct 2026

Analysis of Stochastic Processes through Replay Buffers

arXiv 2206.12848 ↗First posted 26 Jun 2022cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 18 of its 41 references that are in the map. First posted on arXiv 26 Jun 2022.

Builds on

18 of 41 references.

References · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Prioritized Experience Replaycited by 1,099
  6. 2018Distributed Prioritized Experience Replaycited by 269
  7. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  8. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  9. 2020Revisiting Fundamentals of Experience Replaycited by 175
  10. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
  11. 2017A Deeper Look at Experience Replaycited by 105
  12. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
6 more
  1. 2019Experience Replay Optimizationcited by 48
  2. 2017The Effects of Memory Replay in Reinforcement Learningcited by 23
  3. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  4. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  5. 2021Large Batch Experience Replaycited by 14
  6. 2021Sim and Real: Better Togethercited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 1; 2026: 3.

9 papers.

  1. 2022Understanding Deep Neural Function Approximation in Reinforcement Learning via ε-Greedy Explorationcited by 11
  2. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  3. 2026Multiscale replay: A robust algorithm for stochastic variational inequalities with a Markovian buffercited by 2
  4. 2023Finite-Time Analysis of Temporal Difference Learning with Experience Replaycited by 1
  5. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  6. 2023Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropoutcited by 0
  7. 2025Sig2Model: A Boosting-Driven Model for Updatable Learned Indexescited by 0
  8. 2026Beyond the Independence Assumption: Finite-Sample Guarantees for Deep Q-Learning under τ-Mixingcited by 0
  9. 2026A Convergence Framework for Deep V-Learning: Error Propagation and Sharp Action-Gap Boundscited by 0