mlsanityData to 14 Oct 2026

Fast Population-Based Reinforcement Learning on a Single Machine

arXiv 2206.08888 ↗First posted 17 Jun 2022cs.LG cs.AI cs.NEcited by 7

Not on the map. Why?

Abstract (arXiv)

Training populations of agents has demonstrated great promise in Reinforcement Learning for stabilizing training, improving exploration and asymptotic performance, and generating a diverse set of solutions. However, population-based training is often not considered by practitioners as it is perceived to be either prohibitively slow (when implemented sequentially), or computationally expensive (if agents are trained in parallel on independent accelerators). In this work, we compare implementations and revisit previous studies to show that the judicious use of compilation and vectorization allows population-based training to be performed on a single machine with one accelerator with minimal overhead compared to training a single agent. We also show that, when provided with a few accelerators, our protocols extend to large population sizes for applications such as hyperparameter tuning. We hope that this work and the public release of our code will encourage practitioners to use population-based learning more frequently for their research and applications.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

25 of 33 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  12. 2017Population Based Training of Neural Networkscited by 367
13 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  2. 2018Distributed Prioritized Experience Replaycited by 269
  3. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  4. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  5. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  6. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
  7. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  8. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 58
  9. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 54
  10. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  11. 2020Scaling MAP-Elites to Deep Neuroevolutioncited by 38
  12. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  13. 2020Population-Guided Parallel Policy Search for Reinforcement Learningcited by 21

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 1.

7 papers.

  1. 2022Winner Takes It All: Training Performant RL Populations for Combinatorial Optimizationcited by 58
  2. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 29
  3. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 8
  4. 2023Evolving Populations of Diverse RL Agents with MAP-Elitescited by 5
  5. 2024Benchmarking Population-Based Reinforcement Learning across Robotic Tasks with GPU-Accelerated Simulationcited by 1
  6. 2025EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learningcited by 1
  7. 2024Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learningcited by 0