mlsanityData to 14 Oct 2026

Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform

arXiv 2310.00036 ↗First posted 29 Sep 2023cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Distributed Deep Reinforcement Learning (DRL) aims to leverage more computational resources to train autonomous agents with less training time. Despite recent progress in the field, reproducibility issues have not been sufficiently explored. This paper first shows that the typical actor-learner framework can have reproducibility issues even if hyperparameters are controlled. We then introduce Cleanba, a new open-source platform for distributed DRL that proposes a highly reproducible architecture. Cleanba implements highly optimized distributed variants of PPO and IMPALA. Our Atari experiments show that these variants can obtain equivalent or higher scores than strong IMPALA baselines in moolib and torchbeast and PPO baseline in CleanRL. However, Cleanba variants present 1) shorter training time and 2) more reproducible learning curves in different hardware settings. Cleanba's source code is available at urlhttps://github.com/vwxyzjn/cleanba

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

20 of 33 references.

References · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  10. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  11. 2018An Empirical Model of Large-Batch Trainingcited by 281
  12. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
8 more
  1. 2018Distributed Prioritized Experience Replaycited by 269
  2. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  3. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  4. 2016Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPUcited by 48
  5. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  6. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  7. 2021Podracer architectures for scalable Reinforcement Learningcited by 38
  8. 2020High-Throughput Synchronous Deep RLcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 3.

7 papers.

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 29
  4. 2024Planning in a recurrent neural network that plays Sokobancited by 18
  5. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 13
  6. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 2
  7. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1