mlsanityData to 7 Oct 2026

Reverb: A Framework For Experience Replay

arXiv 2102.04736 ↗First posted 9 Feb 2021cs.LG cs.AI cs.DCcited by 27

Not on the map. Why?

Abstract (arXiv)

A central component of training in Reinforcement Learning (RL) is Experience: the data used for training. The mechanisms used to generate and consume this data have an important effect on the performance of RL algorithms. In this paper, we introduce Reverb: an efficient, extensible, and easy to use system designed specifically for experience replay in RL. Reverb is designed to work efficiently in distributed configurations with up to thousands of concurrent clients. The flexible API provides users with the tools to easily and accurately configure the replay buffer. It includes strategies for selecting and removing elements from the buffer, as well as options for controlling the ratio between sampled and inserted elements. This paper presents the core design of Reverb, gives examples of how it can be applied, and provides empirical results of Reverb's performance characteristics.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 10 of its 23 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

10 of 23 references.

References · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2018Distributed Prioritized Experience Replaycited by 260
  7. 2020Revisiting Fundamentals of Experience Replaycited by 171
  8. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  9. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  10. 2017The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 6; 2024: 0; 2025: 3; 2026: 3.

27 papers.

Sort:

By area:

  1. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 40
  2. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  3. 2022Human-level Atari 200x fastercited by 28
  4. 2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
  5. 2021Active Offline Policy Selectioncited by 26
  6. 2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 17
  7. 2023Curious Replay for Model-based Adaptationcited by 16
  8. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  9. 2021Launchpad: A Programming Model for Distributed Machine Learning Researchcited by 14
  10. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  11. 2021Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes (Technical Report)cited by 9
  12. 2022Learning Robust Real-Time Cultural Transmission without Human Datacited by 9
15 more
  1. 2022A Mixture of Surprises for Unsupervised Reinforcement Learningcited by 9
  2. 2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
  3. 2022Value Function Decomposition for Iterative Design of Reinforcement Learning Agentscited by 6
  4. 2022Phantom -- A RL-driven multi-agent framework to model complex systemscited by 5
  5. 2023GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Modelscited by 5
  6. 2022Event Tables for Efficient Experience Replaycited by 4
  7. 2021Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operationscited by 3
  8. 2026A Reference Architecture of Reinforcement Learning Frameworkscited by 3
  9. 2023Zero-Shot Transfer of Haptics-Based Object Insertion Policiescited by 2
  10. 2023marl-jax: Multi-Agent Reinforcement Leaning Frameworkcited by 1
  11. 2025Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphscited by 1
  12. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1
  13. 2023Hierarchical Reinforcement Learning in Complex 3D Environmentscited by 0
  14. 2025Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependenciescited by 0
  15. 2026Conduit: An Experience Data Plane for Distributed Reinforcement Learningcited by 0