mlsanityData to 14 Oct 2026

A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQR

arXiv 2602.18933 ↗First posted 21 Feb 2026eess.SY cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

In this work, we propose a stochastic gradient descent (SGD) framework to design data-driven policy gradient descent algorithms for the linear quadratic regulator problem. Two alternative schemes are considered to estimate the policy gradient from stochastic trajectory data: (i) an indirect online identification based approach, in which the system matrices are first estimated and subsequently used to construct the gradient, and (ii) a direct zeroth-order approach, which approximates the gradient using empirical cost evaluations. In both cases, the resulting gradient estimates are random due to stochasticity in the data, allowing us to use SGD theory to analyze the convergence of the associated policy gradient methods. A key technical step consists of modeling the gradient estimates as suitable stochastic gradient oracles, which, because of the way they are computed, are inherently based. We derive sufficient conditions under which SGD with a biased gradient oracle converges asymptotically to the optimal policy, and leverage these conditions to design the parameters of the gradient estimation schemes. Moreover, we compare the advantages and limitations of the two data-driven gradient estimators. Numerical experiments validate the effectiveness of the proposed methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 42 references that are in the map. First posted on arXiv 21 Feb 2026.

Builds on

24 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025On the Robustness of Derivative-free Methods for Linear Quadratic Regulator14 Jun
  2. 2025Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches6 May
  3. 2025Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data27 Feb
  4. 2024Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lens16 Apr

Other references · 20

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  4. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  5. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  6. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  7. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  8. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  9. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  10. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
  11. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  12. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
8 more
  1. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 35
  2. 2021Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRscited by 16
  3. 2020Robust Policy Iteration for Continuous-time Linear Quadratic Regulationcited by 15
  4. 2022Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guaranteescited by 14
  5. 2020Non-asymptotic bounds for stochastic optimization with biased noisy gradient oraclescited by 6
  6. 2024Stochastic Online Optimization for Cyber-Physical and Robotic Systemscited by 6
  7. 2025Robustness of Online Identification-based Policy Iteration to Noisy Datacited by 3
  8. 2025Perturbed Gradient Descent Algorithms are Small-Disturbance Input-to-State Stablecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression
  2. 2026(Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
  3. 2025Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
  4. 2026On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis
  5. 2024Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint
  6. 2025Power-Constrained Policy Gradient Methods for LQR
  7. 2025Policy Gradient for LQR with Domain Randomization
  8. 2025On the Gradient Domination of the LQG Problem