# Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data arXiv: https://arxiv.org/abs/2502.19977 · HTML: https://mlsanity.com/paper/2502.19977/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. - Builds on 27 of its 51 references that are in the map. - First posted on arXiv 27 Feb 2025. ## Placed in - Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/) ## Builds on (most cited first, 27 of 27) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016) - [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018) - [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020) - [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017) - [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018) - [Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies](https://mlsanity.com/paper/1906.08383/) (2019) - [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020) - [Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems](https://mlsanity.com/paper/1812.08305/) (2018) - [Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon](https://mlsanity.com/paper/2011.10300/) (2020) - [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019) - [Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem](https://mlsanity.com/paper/1912.11899/) (2019) - [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018) - [Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently](https://mlsanity.com/paper/2002.08095/) (2020) - [Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator](https://mlsanity.com/paper/1712.08642/) (2017) - [Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games](https://mlsanity.com/paper/2107.13090/) (2021) - [Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs](https://mlsanity.com/paper/2104.04901/) (2021) - [Thompson Sampling for Linear-Quadratic Control Problems](https://mlsanity.com/paper/1703.08972/) (2017) - [How are policy gradient methods affected by the limits of control?](https://mlsanity.com/paper/2206.06863/) (2022) - [Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lens](https://mlsanity.com/paper/2404.10851/) (2024) - [Convergence and Sample Complexity of Policy Gradient Methods for Stabilizing Linear Systems](https://mlsanity.com/paper/2205.14335/) (2022) - [Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators](https://mlsanity.com/paper/2303.08431/) (2023) - [Structured Output Feedback Control for Linear Quadratic Regulator Using Policy Gradient Method](https://mlsanity.com/paper/2203.00289/) (2022) - [The Role of Identification in Data-driven Policy Iteration: A System Theoretic Study](https://mlsanity.com/paper/2401.06721/) (2024) - [An Efficient Off-Policy Reinforcement Learning Algorithm for the Continuous-Time LQR Problem](https://mlsanity.com/paper/2303.17819/) (2023) - [Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control](https://mlsanity.com/paper/2310.05230/) (2023) - [Data-Driven LQR with Finite-Time Experiments via Extremum-Seeking Policy Iteration](https://mlsanity.com/paper/2412.02758/) (2024) ## Built on by (most cited first, 2 of 2) - [Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression](https://mlsanity.com/paper/2512.03764/) (2025) - [A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQR](https://mlsanity.com/paper/2602.18933/) (2026) ## Abstract (arXiv) Policy gradient (PG) methods are the backbone of many reinforcement learning algorithms due to their good performance in policy optimization problems. As a gradient-based approach, PG methods typically rely on knowledge of the system dynamics. If this is not available, trajectory data can be utilized to approximate first-order information. When the data are noisy, gradient estimates become inaccurate and a study that investigates uncertainty estimation and the analysis of its propagation through the algorithm is currently missing. To address this, our work focuses on the Linear Quadratic Regulator (LQR) problem for systems subject to additive stochastic noise. After briefly summarizing the state of the art for cases with a known model, we focus on scenarios where the system dynamics are unknown, and approximate gradient information is obtained using zeroth-order optimization techniques. We analyze the theoretical properties by computing the error in the estimated gradient and examining how this error affects the convergence of PG algorithms. Additionally, we provide global convergence guarantees for various versions of PG methods, including those employing adaptive step sizes and variance reduction techniques, which help increase the convergence rate and reduce sample complexity. This study contributed to characterizing the robustness of model-free PG methods, aiming to identify their limitations in the presence of stochastic noise and proposing improvements to enhance their applicability.