mlsanityData to 14 Oct 2026

Asynchronous Heterogeneous Linear Quadratic Regulator Design

arXiv 2404.09061 ↗First posted 13 Apr 2024math.OCcited by 4

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

We address the problem of designing an LQR controller in a distributed setting, where M similar but not identical systems share their locally computed policy gradient (PG) estimates with a server that aggregates the estimates and computes a controller that, on average, performs well on all systems. Learning in a distributed setting has the potential to offer statistical benefits - multiple datasets can be leveraged simultaneously to produce more accurate policy gradient estimates. However, the interplay of heterogeneous trajectory data and varying levels of local computational power introduce bias to the aggregated PG descent direction, and prevents us from fully exploiting the parallelism in the distributed computation. The latter stems from synchronous aggregation, where straggler systems negatively impact the runtime. To address this, we propose an asynchronous policy gradient algorithm for LQR control design. By carefully controlling the "staleness" in the asynchronous aggregation, we show that the designed controller converges to each system's ε-near optimal controller up to a heterogeneity bias. Furthermore, we prove that our asynchronous approach obtains exact local convergence at a sub-linear rate.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 35 references that are in the map. First posted on arXiv 13 Apr 2024.

Builds on

26 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQR25 Jan

Other references · 25

  1. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  2. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  3. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  4. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  5. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
  6. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
  7. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 63
  8. 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 46
  9. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  10. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 36
  11. 2022Multi-Task Imitation Learning for Linear Dynamical Systemscited by 23
  12. 2023Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approachcited by 23
13 more
  1. 2015Asynchronous stochastic convex optimizationcited by 19
  2. 2021Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updatescited by 17
  3. 2022FedSysID: A Federated Approach to Sample-Efficient System Identificationcited by 17
  4. 2023Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Datacited by 13
  5. 2022Unbounded Gradients in Federated Learning with Buffered Asynchronous Aggregationcited by 12
  6. 2023Learning Personalized Models with Clustered System Identificationcited by 11
  7. 2022How are policy gradient methods affected by the limits of control?cited by 10
  8. 2023Multi-Task System Identification of Similar Linear Time-Invariant Dynamical Systemscited by 8
  9. 2020Zeroth-Order Feedback Optimization for Cooperative Multi-Agent Systemscited by 7
  10. 2024Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Samplingcited by 7
  11. 2023CSMAAFL: Client Scheduling and Model Aggregation in Asynchronous Federated Learningcited by 2
  12. 2023Oracle Complexity Reduction for Model-free LQR: A Stochastic Variance-Reduced Policy Gradient Approachcited by 2
  13. 2024DASA: Delay-Adaptive Multi-Agent Stochastic Approximationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learningcited by 6
  2. 2025Learning Stabilizing Policies via an Unstable Subspace Representationcited by 3
  3. 2025Policy Gradient Bounds in Multitask LQRcited by 2
  4. 2026Multitask LQG Control: Performance and Generalization Boundscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Asynchronous Parallel Policy Gradient Methods for the Linear Quadratic Regulator
  2. 2025Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning
  3. 2026A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQR
  4. 2025Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches
  5. 2026On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis
  6. 2025On the Gradient Domination of the LQG Problem
  7. 2026(Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
  8. 2025Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data