mlsanityData to 14 Oct 2026

GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Models

arXiv 2310.05205 ↗First posted 8 Oct 2023cs.LG cs.AI cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper introduces a distributed, GPU-centric experience replay system, GEAR, designed to perform scalable reinforcement learning (RL) with large sequence models (such as transformers). With such models, existing systems such as Reverb face considerable bottlenecks in memory, computation, and communication. GEAR, however, optimizes memory efficiency by enabling the memory resources on GPU servers (including host memory and device memory) to manage trajectory data. Furthermore, it facilitates decentralized GPU devices to expedite various trajectory selection strategies, circumventing computational bottlenecks. GEAR is equipped with GPU kernels capable of collecting trajectories using zero-copy access to host memory, along with remote-directed-memory access over InfiniBand, improving communication efficiency. Cluster experiments have shown that GEAR can achieve performance levels up to 6x greater than Reverb when training state-of-the-art large RL models. GEAR is open-sourced at https://github.com/bigrl-team/gear.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

28 of 43 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2022A Generalist Agentcited by 808
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
16 more
  1. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
  2. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  3. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  4. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  5. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  6. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  7. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  8. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  9. 2019rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorchcited by 38
  10. 2021Reverb: A Framework For Experience Replaycited by 27
  11. 2022TorchOpt: An Efficient Library for Differentiable Optimizationcited by 17
  12. 2019CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Serverscited by 14
  13. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 12
  14. 2023Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awarenesscited by 11
  15. 2022On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspectivecited by 8
  16. 2020Efficient Reinforcement Learning Development with RLzoocited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  3. 2025Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphscited by 1
  4. 2026BatchGen: An Architecture for Scalable and Efficient Batch Inferencecited by 0
  5. 2026Conduit: An Experience Data Plane for Distributed Reinforcement Learningcited by 0