# Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control

arXiv: https://arxiv.org/abs/2006.09178 · HTML: https://mlsanity.com/paper/2006.09178/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 13 of 13)

- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise](https://mlsanity.com/paper/2408.02489/) (2024)
- [Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States](https://mlsanity.com/paper/2402.07875/) (2024)
- [Scalable regret for learning to control network-coupled subsystems with unknown dynamics](https://mlsanity.com/paper/2108.07970/) (2021)
- [Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games](https://mlsanity.com/paper/2506.05894/) (2025)
- [Gradient Dominance in the Linear Quadratic Regulator: A Unified Analysis for Continuous-Time and Discrete-Time Systems](https://mlsanity.com/paper/2602.22577/) (2026)
- [Regret Analysis of Policy Optimization over Submanifolds for Linearly Constrained Online LQG](https://mlsanity.com/paper/2403.08553/) (2024)
- [Convergence of Policy Gradient for Stochastic Linear-Quadratic Control Problem in Infinite Horizon](https://mlsanity.com/paper/2404.11382/) (2024)
- [Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error](https://mlsanity.com/paper/2506.09685/) (2025)
- [Some remarks on gradient dominance and LQR policy optimization](https://mlsanity.com/paper/2507.10452/) (2025)
- [Data-Driven Continuous-Time Linear Quadratic Regulator via Closed-Loop and Reinforcement Learning Parameterizations](https://mlsanity.com/paper/2604.27922/) (2026)

## Abstract (arXiv)

We consider the continuous-time Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains. The results developed are in parallel to those in Bu et al. [1] for discrete-time LTI systems. In this direction, we characterize several analytical properties (smoothness, coerciveness, quadratic growth) that are crucial in the analysis of gradient-based algorithms. We also point out similarities and distinctive features of the continuous time setup in comparison with its discrete time analogue. First, we examine three types of well-posed flows direct policy update for LQR: gradient flow, natural gradient flow and the quasi-Newton flow. The coercive property of the corresponding cost function suggests that these flows admit unique solutions while the gradient dominated property indicates that the underling Lyapunov functionals decay at an exponential rate; quadratic growth on the other hand guarantees that the trajectories of these flows are exponentially stable in the sense of Lyapunov. We then discuss the forward Euler discretization of these flows, realized as gradient descent, natural gradient descent and quasi-Newton iteration. We present stepsize criteria for gradient descent and natural gradient descent, guaranteeing that both algorithms converge linearly to the global optima. An optimal stepsize for the quasi-Newton iteration is also proposed, guaranteeing a Q-quadratic convergence rate--and in the meantime--recovering the Kleinman-Newton iteration. Lastly, we examine LQR state feedback synthesis with a sparsity pattern. In this case, we develop the necessary formalism and insights for projected gradient descent, allowing us to guarantee a sublinear rate of convergence to a first-order stationary point.