mlsanityData to 14 Oct 2026

Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data

arXiv 2502.19977 ↗First posted 27 Feb 2025eess.SY cs.SYcited by 2

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Policy gradient (PG) methods are the backbone of many reinforcement learning algorithms due to their good performance in policy optimization problems. As a gradient-based approach, PG methods typically rely on knowledge of the system dynamics. If this is not available, trajectory data can be utilized to approximate first-order information. When the data are noisy, gradient estimates become inaccurate and a study that investigates uncertainty estimation and the analysis of its propagation through the algorithm is currently missing. To address this, our work focuses on the Linear Quadratic Regulator (LQR) problem for systems subject to additive stochastic noise. After briefly summarizing the state of the art for cases with a known model, we focus on scenarios where the system dynamics are unknown, and approximate gradient information is obtained using zeroth-order optimization techniques. We analyze the theoretical properties by computing the error in the estimated gradient and examining how this error affects the convergence of PG algorithms. Additionally, we provide global convergence guarantees for various versions of PG methods, including those employing adaptive step sizes and variance reduction techniques, which help increase the convergence rate and reduce sample complexity. This study contributed to characterizing the robustness of model-free PG methods, aiming to identify their limitations in the presence of stochastic noise and proposing improvements to enhance their applicability.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 51 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

27 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lens16 Apr

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  4. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  5. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
  6. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  7. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  8. 2020Naive Exploration is Optimal for Online LQRcited by 116
  9. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
  10. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  11. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
  12. 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 46
14 more
  1. 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 40
  2. 2020Logarithmic Regret for Learning Linear Quadratic Regulators Efficientlycited by 35
  3. 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 30
  4. 2021Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Gamescited by 25
  5. 2021Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRscited by 16
  6. 2017Thompson Sampling for Linear-Quadratic Control Problemscited by 14
  7. 2022How are policy gradient methods affected by the limits of control?cited by 10
  8. 2022Convergence and Sample Complexity of Policy Gradient Methods for Stabilizing Linear Systemscited by 6
  9. 2023Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulatorscited by 5
  10. 2022Structured Output Feedback Control for Linear Quadratic Regulator Using Policy Gradient Methodcited by 3
  11. 2024The Role of Identification in Data-driven Policy Iteration: A System Theoretic Studycited by 3
  12. 2023An Efficient Off-Policy Reinforcement Learning Algorithm for the Continuous-Time LQR Problemcited by 2
  13. 2023Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Controlcited by 2
  14. 2024Data-Driven LQR with Finite-Time Experiments via Extremum-Seeking Policy Iterationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regressioncited by 0
  2. 2026A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026(Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
  2. 2025Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches
  3. 2024Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint
  4. 2025On the Robustness of Derivative-free Methods for Linear Quadratic Regulator
  5. 2025Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
  6. 2024Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise
  7. 2024Convergence of Policy Gradient for Stochastic Linear-Quadratic Control Problem in Infinite Horizon
  8. 2025On the Gradient Domination of the LQG Problem