# Efficient Parallel Methods for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1705.04862 · HTML: https://mlsanity.com/paper/1705.04862/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2017.

## Built on by (most cited first, 30 of 32)

- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2108.11887/) (2021)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation](https://mlsanity.com/paper/2103.04972/) (2021)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Communication Efficient Parallel Reinforcement Learning](https://mlsanity.com/paper/2102.10740/) (2021)
- [Cooperative Multi-Agent Reinforcement Learning: Asynchronous Communication and Linear Function Approximation](https://mlsanity.com/paper/2305.06446/) (2023)
- [Social Contract AI: Aligning AI Assistants with Implicit Group Norms](https://mlsanity.com/paper/2310.17769/) (2023)
- [A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective](https://mlsanity.com/paper/2311.16065/) (2023)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2208.12328/) (2022)
- [Rate-Constrained Remote Contextual Bandits](https://mlsanity.com/paper/2204.12620/) (2022)
- [Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox](https://mlsanity.com/paper/2212.00253/) (2022)
- [Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story](https://mlsanity.com/paper/2505.01336/) (2025)
- [Remote Contextual Bandits](https://mlsanity.com/paper/2202.05182/) (2022)
- [Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations](https://mlsanity.com/paper/2110.01101/) (2021)
- [MAN: Multi-Action Networks Learning](https://mlsanity.com/paper/2209.09329/) (2022)
- [SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systems](https://mlsanity.com/paper/2405.03967/) (2024)
- [Scaling up Stochastic Gradient Descent for Non-convex Optimisation](https://mlsanity.com/paper/2210.02882/) (2022)
- [Speeding up Policy Simulation in Supply Chain RL](https://mlsanity.com/paper/2406.01939/) (2024)
- [Design Optimization of Nuclear Fusion Reactor through Deep Reinforcement Learning](https://mlsanity.com/paper/2409.08231/) (2024)
- [Toward Debugging Deep Reinforcement Learning Programs with RLExplorer](https://mlsanity.com/paper/2410.04322/) (2024)
- [Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04883/) (2025)
- [Human-Level Control without Server-Grade Hardware](https://mlsanity.com/paper/2111.01264/) (2021)
- [Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies](https://mlsanity.com/paper/2202.00360/) (2022)
- [Long N-step Surrogate Stage Reward to Reduce Variances of Deep Reinforcement Learning in Complex Problems](https://mlsanity.com/paper/2210.04820/) (2022)
- [Parallel Q-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation](https://mlsanity.com/paper/2307.12983/) (2023)
- [A Review of Deep Reinforcement Learning in Serverless Computing: Function Scheduling and Resource Auto-Scaling](https://mlsanity.com/paper/2311.12839/) (2023)

## Abstract (arXiv)

We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at https://github.com/alfredvc/paac