# Second-Order Policy Gradient Methods for the Linear Quadratic Regulator

arXiv: https://arxiv.org/abs/2511.02095 · HTML: https://mlsanity.com/paper/2511.02095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 8 of 8)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Global Optimality Guarantees For Policy Gradient Methods](https://mlsanity.com/paper/1906.01786/) (2019)
- [Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time](https://mlsanity.com/paper/2008.06845/) (2020)
- [A Family of Iterative Gauss-Newton Shooting Methods for Nonlinear Optimal Control](https://mlsanity.com/paper/1711.11006/) (2017)
- [Quasi-Newton Iteration in Deterministic Policy Gradient](https://mlsanity.com/paper/2203.13854/) (2022)
- [Convergence and Robustness of Value and Policy Iteration for the Linear Quadratic Regulator](https://mlsanity.com/paper/2411.04548/) (2024)

## Abstract (arXiv)

Policy gradient methods are a powerful family of reinforcement learning algorithms for continuous control that optimize a policy directly. However, standard first-order methods often converge slowly. Second-order methods can accelerate learning by using curvature information, but they are typically expensive to compute. The linear quadratic regulator (LQR) is a practical setting in which key quantities, such as the policy gradient, admit closed-form expressions. In this work, we develop second-order policy gradient algorithms for LQR by deriving explicit formulas for both the approximate and exact Hessians used in Gauss--Newton and Newton methods, respectively. Numerical experiments show a faster convergence rate for the proposed second-order approach over the standard first-order policy gradient baseline.