# Reverb: A Framework For Experience Replay

arXiv: https://arxiv.org/abs/2102.04736 · HTML: https://mlsanity.com/paper/2102.04736/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Diagnosing Bottlenecks in Deep Q-learning Algorithms](https://mlsanity.com/paper/1902.10250/) (2019)
- [The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning](https://mlsanity.com/paper/1704.04651/) (2017)

## Built on by (most cited first, 27 of 27)

- [Understanding Self-Predictive Learning for Reinforcement Learning](https://mlsanity.com/paper/2212.03319/) (2022)
- [Laminar: A Scalable Asynchronous RL Post-Training Framework](https://mlsanity.com/paper/2510.12633/) (2025)
- [Human-level Atari 200x faster](https://mlsanity.com/paper/2209.07550/) (2022)
- [Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding](https://mlsanity.com/paper/2312.05328/) (2023)
- [Active Offline Policy Selection](https://mlsanity.com/paper/2106.10251/) (2021)
- [Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization](https://mlsanity.com/paper/2207.13676/) (2022)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Evaluating model-based planning and planner amortization for continuous control](https://mlsanity.com/paper/2110.03363/) (2021)
- [Launchpad: A Programming Model for Distributed Machine Learning Research](https://mlsanity.com/paper/2106.04516/) (2021)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes (Technical Report)](https://mlsanity.com/paper/2112.09655/) (2021)
- [Learning Robust Real-Time Cultural Transmission without Human Data](https://mlsanity.com/paper/2203.00715/) (2022)
- [A Mixture of Surprises for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2210.06702/) (2022)
- [MAD for Robust Reinforcement Learning in Machine Translation](https://mlsanity.com/paper/2207.08583/) (2022)
- [Value Function Decomposition for Iterative Design of Reinforcement Learning Agents](https://mlsanity.com/paper/2206.13901/) (2022)
- [Phantom -- A RL-driven multi-agent framework to model complex systems](https://mlsanity.com/paper/2210.06012/) (2022)
- [GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Models](https://mlsanity.com/paper/2310.05205/) (2023)
- [Event Tables for Efficient Experience Replay](https://mlsanity.com/paper/2211.00576/) (2022)
- [Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operations](https://mlsanity.com/paper/2106.11345/) (2021)
- [A Reference Architecture of Reinforcement Learning Frameworks](https://mlsanity.com/paper/2603.06413/) (2026)
- [Zero-Shot Transfer of Haptics-Based Object Insertion Policies](https://mlsanity.com/paper/2301.12587/) (2023)
- [Rollout-Level Advantage-Prioritized Experience Replay for GRPO](https://mlsanity.com/paper/2606.04560/) (2026)
- [marl-jax: Multi-Agent Reinforcement Leaning Framework](https://mlsanity.com/paper/2303.13808/) (2023)
- [Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs](https://mlsanity.com/paper/2501.05408/) (2025)
- [Hierarchical Reinforcement Learning in Complex 3D Environments](https://mlsanity.com/paper/2302.14451/) (2023)
- [Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies](https://mlsanity.com/paper/2502.20190/) (2025)
- [Conduit: An Experience Data Plane for Distributed Reinforcement Learning](https://mlsanity.com/paper/2609.24456/) (2026)

## Abstract (arXiv)

A central component of training in Reinforcement Learning (RL) is Experience: the data used for training. The mechanisms used to generate and consume this data have an important effect on the performance of RL algorithms. In this paper, we introduce Reverb: an efficient, extensible, and easy to use system designed specifically for experience replay in RL. Reverb is designed to work efficiently in distributed configurations with up to thousands of concurrent clients. The flexible API provides users with the tools to easily and accurately configure the replay buffer. It includes strategies for selecting and removing elements from the buffer, as well as options for controlling the ratio between sampled and inserted elements. This paper presents the core design of Reverb, gives examples of how it can be applied, and provides empirical results of Reverb's performance characteristics.