# LQR through the Lens of First Order Methods: Discrete-time Case

arXiv: https://arxiv.org/abs/1907.08921 · HTML: https://mlsanity.com/paper/1907.08921/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2019.

## Built on by (most cited first, 30 of 40)

- [A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces](https://mlsanity.com/paper/2310.02951/) (2023)
- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQR](https://mlsanity.com/paper/2401.14534/) (2024)
- [Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity](https://mlsanity.com/paper/2101.01041/) (2021)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradient](https://mlsanity.com/paper/2302.13144/) (2023)
- [On the Sample Complexity of Decentralized Linear Quadratic Regulator with Partially Nested Information Structure](https://mlsanity.com/paper/2110.07112/) (2021)
- [Global Convergence of Direct Policy Search for State-Feedback H_∞ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential](https://mlsanity.com/paper/2210.11577/) (2022)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Model-Free μ Synthesis via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2111.15537/) (2021)
- [From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent](https://mlsanity.com/paper/2210.06705/) (2022)
- [Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators](https://mlsanity.com/paper/2303.08431/) (2023)
- [Convergence of Gradient-based MAML in LQR](https://mlsanity.com/paper/2309.06588/) (2023)
- [Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches](https://mlsanity.com/paper/2505.03706/) (2025)
- [Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent](https://mlsanity.com/paper/2107.12416/) (2021)
- [Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedback](https://mlsanity.com/paper/2310.19022/) (2023)
- [Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States](https://mlsanity.com/paper/2402.07875/) (2024)
- [Learning Stochastic Optimal Policies via Gradient Descent](https://mlsanity.com/paper/2106.03780/) (2021)
- [Scalable Reinforcement Learning for Linear-Quadratic Control of Networks](https://mlsanity.com/paper/2401.16183/) (2024)
- [On the Complexity of Finite-Sum Smooth Optimization under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/2402.02569/) (2024)
- [Linear Dynamics meets Linear MDPs: Closed-Form Optimal Policies via Reinforcement Learning](https://mlsanity.com/paper/2508.17185/) (2025)
- [On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator](https://mlsanity.com/paper/2209.05042/) (2022)
- [Learning Decentralized Linear Quadratic Regulators with √T Regret](https://mlsanity.com/paper/2210.08886/) (2022)
- [Distributed Policy Gradient for Linear Quadratic Networked Control with Limited Communication Range](https://mlsanity.com/paper/2403.03055/) (2024)
- [Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient](https://mlsanity.com/paper/2403.05367/) (2024)
- [Direct Data-Driven Discounted Infinite Horizon Linear Quadratic Regulator with Robustness Guarantees](https://mlsanity.com/paper/2409.10703/) (2024)
- [Policy Gradient Method for LQG Control via Input-Output-History Representation: Convergence to O(ε)-Stationary Points](https://mlsanity.com/paper/2510.19141/) (2025)
- [Gradient Dominance in the Linear Quadratic Regulator: A Unified Analysis for Continuous-Time and Discrete-Time Systems](https://mlsanity.com/paper/2602.22577/) (2026)
- [Accelerated Optimization Landscape of Linear-Quadratic Regulator](https://mlsanity.com/paper/2307.03590/) (2023)

## Abstract (arXiv)

We consider the Linear-Quadratic-Regulator (LQR) problem in terms of optimizing a real-valued matrix function over the set of feedback gains. Such a setup facilitates examining the implications of a natural initial-state independent formulation of LQR in designing first order algorithms. It is shown that this cost function is smooth and coercive, and provide an alternate means of noting its gradient dominated property. In the process, we provide a number of analytic observations on the LQR cost when directly analyzed in terms of the feedback gain. We then examine three types of well-posed flows for LQR: gradient flow, natural gradient flow and the quasi-Newton flow. The coercive property suggests that these flows admit unique solutions while gradient dominated property indicates that the corresponding Lyapunov functionals decay at an exponential rate; we also prove that these flows are exponentially stable in the sense of Lyapunov. We then discuss the forward Euler discretization of these flows, realized as gradient descent, natural gradient descent and the quasi-Newton iteration. We present stepsize criteria for gradient descent and natural gradient descent, guaranteeing that both algorithms converge linearly to the global optima. An optimal stepsize for the quasi-Newton iteration is also proposed, guaranteeing a Q-quadratic convergence rate--and in the meantime--recovering the Hewer algorithm.