mlsanityData to 14 Oct 2026

Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

arXiv 2411.05614 ↗First posted 8 Nov 2024cs.LG cs.AI cs.DCcited by 8

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning has led to dramatic breakthroughs in the field of artificial intelligence for the past few years. As the amount of rollout experience data and the size of neural networks for deep reinforcement learning have grown continuously, handling the training process and reducing the time consumption using parallel and distributed computing is becoming an urgent and essential desire. In this paper, we perform a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references. In particular, a taxonomy of literature is provided, along with a discussion of emerging topics and open issues. This incorporates learning system architectures, simulation parallelism, computing parallelism, distributed synchronization mechanisms, and deep evolutionary reinforcement learning. Further, we compare 16 current open-source libraries and platforms with criteria of facilitating rapid development. Finally, we extrapolate future directions that deserve further research.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 102 of its 188 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

102 of 188 references.

References · 102

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
38 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  3. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  8. 2018Learning Dexterous In-Hand Manipulationcited by 687
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  12. 2017Progressive Neural Architecture Searchcited by 487
  13. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  14. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  15. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  16. 2017Large-Scale Evolution of Image Classifierscited by 348
  17. 2017Deep Reinforcement Learning: An Overviewcited by 333
  18. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  19. 2018Distributed Prioritized Experience Replaycited by 269
  20. 2017Noisy Networks for Explorationcited by 266
  21. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  22. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  23. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  24. 2014Robots that can adapt like animalscited by 204
  25. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  26. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 195
  27. 2019A Survey on Distributed Machine Learningcited by 188
  28. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  29. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  30. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  31. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  32. 2021Embodied Intelligence via Learning and Evolutioncited by 163
  33. 2017Parameter Space Noise for Explorationcited by 161
  34. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  35. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  36. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  37. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  38. 2017Revisiting Distributed Synchronous SGDcited by 142

50 most cited of 102 All 102 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 13
  2. 2026A Reference Architecture of Reinforcement Learning Frameworkscited by 3
  3. 2025Large-scale automatic carbon ion treatment planning for head and neck cancers via parallel multi-agent reinforcement learningcited by 1
  4. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
  5. 2025Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependenciescited by 0
  6. 2025Adaptive Policy Synchronization for Scalable Reinforcement Learningcited by 0
  7. 2026Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamicscited by 0
  8. 2026Conduit: An Experience Data Plane for Distributed Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  2. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  3. 2025A Practical Introduction to Deep Reinforcement Learning
  4. 2026Deep Reinforcement Learning: From First Principles to Reasoning Models ↗
  5. 2025Deep Reinforcement Learning in Applied Control: Challenges, Analysis, and Insights
  6. 2024Deep Reinforcement Learning for Dynamic Algorithm Selection: A Proof-of-Principle Study on Differential Evolution
  7. 2024Going into Orbit: Massively Parallelizing Episodic Reinforcement Learning
  8. 2024Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks