Asynchronous Parallel Policy Gradient Methods for the Linear Quadratic Regulator
On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization
Abstract (arXiv)
Learning policies in an asynchronous parallel way is essential to the numerous successes of RL for solving large-scale problems. However, their convergence performance is still not rigorously evaluated. To this end, we adopt the asynchronous parallel zero-order policy gradient (AZOPG) method to solve the continuous-time linear quadratic regulation problem. Specifically, as in the celebrated A3C algorithm, there are multiple parallel workers to asynchronously estimate PGs which are then sent to a central master for policy updates. Via quantifying its convergence rate of policy iterations, we show the linear speedup property of the AZOPG, both in theory and simulation, which clearly reveals the advantages of using parallel workers for learning policies.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 35 references that are in the map. First posted on arXiv 3 Jul 2024.
Builds on
19 of 35 references.
References · 19
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
- 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
- 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
- 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
- 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
- 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 46
- 2016A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Ordercited by 44
- 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 42
- 2019Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approachcited by 38
7 more
- 2020Optimizing Static Linear Feedback: Gradient Methodcited by 27
- 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 25
- 2023Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approachcited by 23
- 2021Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRscited by 16
- 2018AsySPA: An Exact Asynchronous Algorithm for Convex Optimization Over Digraphscited by 7
- 2022Convergence and Sample Complexity of Policy Gradient Methods for Stabilizing Linear Systemscited by 6
- 2020LQR with Tracking: A Zeroth-order Approach and Its Global Convergencecited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Asynchronous Heterogeneous Linear Quadratic Regulator Design
- 2025Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning
- 2025On the Gradient Domination of the LQG Problem
- 2026Global Convergence of Policy Gradient Methods for ReLU Controllers in Linear Quadratic Regulation
- 2026Align and Filter: Improving Performance in Asynchronous On-Policy RL
- 2025Convergence of Flow-Policy Gradient Learning for Linear Quadratic Regulator Problems
- 2026(Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
- 2025Second-Order Policy Gradient Methods for the Linear Quadratic Regulator