# Analysis of Stochastic Processes through Replay Buffers

arXiv: https://arxiv.org/abs/2206.12848 · HTML: https://mlsanity.com/paper/2206.12848/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 26 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [A Finite Time Analysis of Two Time-Scale Actor Critic Methods](https://mlsanity.com/paper/2005.01350/) (2020)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Finite-Sample Analysis for SARSA with Linear Function Approximation](https://mlsanity.com/paper/1902.02234/) (2019)
- [Experience Replay Optimization](https://mlsanity.com/paper/1906.08387/) (2019)
- [The Effects of Memory Replay in Reinforcement Learning](https://mlsanity.com/paper/1710.06574/) (2017)
- [Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains](https://mlsanity.com/paper/1806.04624/) (2018)
- [Improved Regret Bound and Experience Replay in Regularized Policy Iteration](https://mlsanity.com/paper/2102.12611/) (2021)
- [Large Batch Experience Replay](https://mlsanity.com/paper/2110.01528/) (2021)
- [Sim and Real: Better Together](https://mlsanity.com/paper/2110.00445/) (2021)

## Built on by (most cited first, 9 of 9)

- [Understanding Deep Neural Function Approximation in Reinforcement Learning via ε-Greedy Exploration](https://mlsanity.com/paper/2209.07376/) (2022)
- [Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning](https://mlsanity.com/paper/2302.11147/) (2023)
- [Multiscale replay: A robust algorithm for stochastic variational inequalities with a Markovian buffer](https://mlsanity.com/paper/2601.01502/) (2026)
- [Finite-Time Analysis of Temporal Difference Learning with Experience Replay](https://mlsanity.com/paper/2306.09746/) (2023)
- [Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences](https://mlsanity.com/paper/2405.14629/) (2024)
- [Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropout](https://mlsanity.com/paper/2301.11168/) (2023)
- [Sig2Model: A Boosting-Driven Model for Updatable Learned Indexes](https://mlsanity.com/paper/2509.20781/) (2025)
- [Beyond the Independence Assumption: Finite-Sample Guarantees for Deep Q-Learning under τ-Mixing](https://mlsanity.com/paper/2605.06373/) (2026)
- [A Convergence Framework for Deep V-Learning: Error Propagation and Sharp Action-Gap Bounds](https://mlsanity.com/paper/2609.18782/) (2026)

## Abstract (arXiv)

Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.