mlsanityData to 14 Oct 2026

Second-Order Policy Gradient Methods for the Linear Quadratic Regulator

arXiv 2511.02095 ↗First posted 3 Nov 2025eess.SY cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Policy gradient methods are a powerful family of reinforcement learning algorithms for continuous control that optimize a policy directly. However, standard first-order methods often converge slowly. Second-order methods can accelerate learning by using curvature information, but they are typically expensive to compute. The linear quadratic regulator (LQR) is a practical setting in which key quantities, such as the policy gradient, admit closed-form expressions. In this work, we develop second-order policy gradient algorithms for LQR by deriving explicit formulas for both the approximate and exact Hessians used in Gauss--Newton and Newton methods, respectively. Numerical experiments show a faster convergence rate for the proposed second-order approach over the standard first-order policy gradient baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 27 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

8 of 27 references.

References · 8

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  4. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  5. 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
  6. 2017A Family of Iterative Gauss-Newton Shooting Methods for Nonlinear Optimal Controlcited by 4
  7. 2022Quasi-Newton Iteration in Deterministic Policy Gradientcited by 3
  8. 2024Convergence and Robustness of Value and Policy Iteration for the Linear Quadratic Regulatorcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression
  2. 2025Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches
  3. 2026A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQR
  4. 2025Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data
  5. 2026(Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
  6. 2025On the Gradient Domination of the LQG Problem
  7. 2025Policy Gradient with Second Order Momentum
  8. 2025Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error