# Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU

arXiv: https://arxiv.org/abs/1611.06256 · HTML: https://mlsanity.com/paper/1611.06256/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2016.

## Built on by (most cited first, 30 of 48)

- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2106.07551/) (2021)
- [Lifelong Reinforcement Learning with Modulating Masks](https://mlsanity.com/paper/2212.11110/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Reinforcing Language Agents via Policy Optimization with Action Decomposition](https://mlsanity.com/paper/2405.15821/) (2024)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)
- [Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks](https://mlsanity.com/paper/2507.23172/) (2025)
- [Agent-Centric Relation Graph for Object Visual Navigation](https://mlsanity.com/paper/2111.14422/) (2021)
- [Benchmarking Quantum Reinforcement Learning](https://mlsanity.com/paper/2501.15893/) (2025)
- [Robust Knowledge Adaptation for Dynamic Graph Neural Networks](https://mlsanity.com/paper/2207.10839/) (2022)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09650/) (2021)
- [Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform](https://mlsanity.com/paper/2310.00036/) (2023)
- [Deep Reinforcement Learning for Conservation Decisions](https://mlsanity.com/paper/2106.08272/) (2021)
- [Costly Features Classification using Monte Carlo Tree Search](https://mlsanity.com/paper/2102.07073/) (2021)
- [Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox](https://mlsanity.com/paper/2212.00253/) (2022)
- [Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education](https://mlsanity.com/paper/2309.13059/) (2023)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision](https://mlsanity.com/paper/2102.13565/) (2021)
- [A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2312.15474/) (2023)
- [Coding for Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2101.02308/) (2021)
- [A Bibliometric Analysis and Review on Reinforcement Learning for Transportation Applications](https://mlsanity.com/paper/2210.14524/) (2022)
- [Deep Reinforcement Learning for ESG financial portfolio management](https://mlsanity.com/paper/2307.09631/) (2023)
- [An advantage actor-critic algorithm for robotic motion planning in dense and dynamic scenarios](https://mlsanity.com/paper/2102.03138/) (2021)
- [Ensemble Consensus-based Representation Deep Reinforcement Learning for Hybrid FSO/RF Communication Systems](https://mlsanity.com/paper/2108.02551/) (2021)
- [Efficient Distributed Framework for Collaborative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.05248/) (2022)
- [Scaling up Stochastic Gradient Descent for Non-convex Optimisation](https://mlsanity.com/paper/2210.02882/) (2022)
- [WILD-SCAV: Benchmarking FPS Gaming AI on Unity3D-based Environments](https://mlsanity.com/paper/2210.09026/) (2022)

## Abstract (arXiv)

We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU's computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C .