# Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

arXiv: https://arxiv.org/abs/2011.10300 · HTML: https://mlsanity.com/paper/2011.10300/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 30 of 50)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Statistical Learning Theory for Control: A Finite Sample Perspective](https://mlsanity.com/paper/2209.05423/) (2022)
- [Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games](https://mlsanity.com/paper/2107.13090/) (2021)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [A Small Gain Analysis of Single Timescale Actor Critic](https://mlsanity.com/paper/2203.02591/) (2022)
- [Reinforcement Learning with Dynamic Convex Risk Measures](https://mlsanity.com/paper/2112.13414/) (2021)
- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity](https://mlsanity.com/paper/2101.01041/) (2021)
- [Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delay](https://mlsanity.com/paper/2307.11436/) (2023)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with √T Regret](https://mlsanity.com/paper/2102.12608/) (2021)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradient](https://mlsanity.com/paper/2302.13144/) (2023)
- [Learning the Kalman Filter with Fine-Grained Sample Complexity](https://mlsanity.com/paper/2301.12624/) (2023)
- [Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods](https://mlsanity.com/paper/2310.02671/) (2023)
- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [From Deep Learning to LLMs: A survey of AI in Quantitative Investment](https://mlsanity.com/paper/2503.21422/) (2025)
- [Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methods](https://mlsanity.com/paper/2109.06362/) (2021)
- [Linear convergence of a policy gradient method for some finite horizon continuous time control problems](https://mlsanity.com/paper/2203.11758/) (2022)
- [Global Convergence of Direct Policy Search for State-Feedback H_∞ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential](https://mlsanity.com/paper/2210.11577/) (2022)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators](https://mlsanity.com/paper/2303.08431/) (2023)
- [Continuous-time stochastic gradient descent for optimizing over the stationary distribution of stochastic differential equations](https://mlsanity.com/paper/2202.06637/) (2022)
- [Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity and Last-Iterate Convergence](https://mlsanity.com/paper/2309.04272/) (2023)
- [Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designs](https://mlsanity.com/paper/2309.04831/) (2023)
- [Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedback](https://mlsanity.com/paper/2310.19022/) (2023)
- [Why Policy Gradient Algorithms Work for Undiscounted Total-Reward MDPs](https://mlsanity.com/paper/2510.18340/) (2025)
- [Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network](https://mlsanity.com/paper/2206.14267/) (2022)

## Abstract (arXiv)

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular, we consider the convergence of policy gradient methods in the setting of known and unknown parameters. We are able to produce a global linear convergence guarantee for this approach in the setting of finite time horizon and stochastic state dynamics under weak assumptions. The convergence of a projected policy gradient method is also established in order to handle problems with constraints. We illustrate the performance of the algorithm with two examples. The first example is the optimal liquidation of a holding in an asset. We show results for the case where we assume a model for the underlying dynamics and where we apply the method to the data directly. The empirical evidence suggests that the policy gradient method can learn the global optimal solution for a larger class of stochastic systems containing the LQR framework and that it is more robust with respect to model mis-specification when compared to a model-based approach. The second example is an LQR system in a higher dimensional setting with synthetic data.