# Accelerated Methods for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1803.02811 · HTML: https://mlsanity.com/paper/1803.02811/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2018.

## Built on by (most cited first, 30 of 40)

- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [Scaling Scaling Laws with Board Games](https://mlsanity.com/paper/2104.03113/) (2021)
- [XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX](https://mlsanity.com/paper/2312.12044/) (2023)
- [Training Larger Networks for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.07920/) (2021)
- [Podracer architectures for scalable Reinforcement Learning](https://mlsanity.com/paper/2104.06272/) (2021)
- [Small batch deep reinforcement learning](https://mlsanity.com/paper/2310.03882/) (2023)
- [MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2106.07551/) (2021)
- [Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size](https://mlsanity.com/paper/2211.11092/) (2022)
- [CFlowNets: Continuous Control with Generative Flow Networks](https://mlsanity.com/paper/2303.02430/) (2023)
- [Learning Sim-to-Real Humanoid Locomotion in 15 Minutes](https://mlsanity.com/paper/2512.01996/) (2025)
- [The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks](https://mlsanity.com/paper/2506.03404/) (2025)
- [Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari](https://mlsanity.com/paper/2111.10247/) (2021)
- [SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores](https://mlsanity.com/paper/2306.16688/) (2023)
- [minimax: Efficient Baselines for Autocurricula in JAX](https://mlsanity.com/paper/2311.12716/) (2023)
- [Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training](https://mlsanity.com/paper/2110.08956/) (2021)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Fast Population-Based Reinforcement Learning on a Single Machine](https://mlsanity.com/paper/2206.08888/) (2022)
- [Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform](https://mlsanity.com/paper/2310.00036/) (2023)
- [Graph augmented Deep Reinforcement Learning in the GameRLand3D environment](https://mlsanity.com/paper/2112.11731/) (2021)
- [Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second](https://mlsanity.com/paper/2306.07552/) (2023)
- [Godot Reinforcement Learning Agents](https://mlsanity.com/paper/2112.03636/) (2021)
- [Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox](https://mlsanity.com/paper/2212.00253/) (2022)
- [Investigating Navigation Strategies in the Morris Water Maze through Deep Reinforcement Learning](https://mlsanity.com/paper/2306.01066/) (2023)
- [Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulation](https://mlsanity.com/paper/2501.18177/) (2025)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Fast and Data Efficient Reinforcement Learning from Pixels via Non-Parametric Value Approximation](https://mlsanity.com/paper/2203.03078/) (2022)
- [MAN: Multi-Action Networks Learning](https://mlsanity.com/paper/2209.09329/) (2022)
- [Actively Learning Costly Reward Functions for Reinforcement Learning](https://mlsanity.com/paper/2211.13260/) (2022)

## Abstract (arXiv)

Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.