Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achieved remarkable empirical success, their distributed counterparts remain less explored, particularly in the presence of heterogeneous asynchronous computations and communication bottlenecks. We introduce two new algorithms, Rennala NIGT and Malenia NIGT, which implement asynchronous policy gradient aggregation and achieve state-of-the-art efficiency. In the homogeneous setting, Rennala NIGT provably improves the total computational and communication complexity while supporting the AllReduce operation. In the heterogeneous setting, Malenia NIGT simultaneously handles asynchronous computations and heterogeneous environments with strictly better theoretical guarantees. Our results are further corroborated by experiments, showing that our methods significantly outperform prior approaches.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 55 references that are in the map. First posted on arXiv 29 Sep 2025.
Builds on
38 of 55 references.
References · 38
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
- 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
- 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
26 more
- 2017Revisiting Distributed Synchronous SGDcited by 142
- 2020Momentum Improves Normalized SGDcited by 139
- 2017Lower Bounds for Finding Stationary Points Icited by 126
- 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
- 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
- 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
- 2021A general sample complexity analysis of vanilla policy gradientcited by 76
- 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
- 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
- 2017Lower Bounds for Finding Stationary Points II: First-Order Methodscited by 58
- 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
- 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
- 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 40
- 2015An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimizationcited by 38
- 2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32
- 2020Momentum-Based Policy Gradient Methodscited by 30
- 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
- 2023Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Modelcited by 24
- 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
- 2025Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexitycited by 18
- 2022Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrizationcited by 17
- 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
- 2024Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneitycited by 13
- 2022Delay-adaptive step-sizes for asynchronous learningcited by 11
- 2024On the Optimal Time Complexities in Decentralized Stochastic Asynchronous Optimizationcited by 8
- 2024Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computationscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learning
- 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
- 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
- 2025Universal Reinforcement Learning in Coalgebras: Asynchronous Stochastic Computation via Conduction
- 2024Asynchronous Stochastic Approximation with Applications to Average-Reward Reinforcement Learning
- 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
- 2024Asynchronous Parallel Policy Gradient Methods for the Linear Quadratic Regulator
- 2024Asynchronous Heterogeneous Linear Quadratic Regulator Design