# How are policy gradient methods affected by the limits of control?

arXiv: https://arxiv.org/abs/2206.06863 · HTML: https://mlsanity.com/paper/2206.06863/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 21 of 21)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies](https://mlsanity.com/paper/1906.08383/) (2019)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019)
- [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018)
- [On the Sample Complexity of Stability Constrained Imitation Learning](https://mlsanity.com/paper/2102.09161/) (2021)
- [Analysis of the Optimization Landscape of Linear Quadratic Gaussian (LQG) Control](https://mlsanity.com/paper/2102.04393/) (2021)
- [Single Trajectory Nonparametric Learning of Nonlinear Dynamics](https://mlsanity.com/paper/2202.08311/) (2022)
- [Regret Bounds for Adaptive Nonlinear Control](https://mlsanity.com/paper/2011.13101/) (2020)
- [Stabilizing Dynamical Systems via Policy Gradient Methods](https://mlsanity.com/paper/2110.06418/) (2021)
- [Linear Systems can be Hard to Learn](https://mlsanity.com/paper/2104.01120/) (2021)
- [Sample Complexity of Linear Quadratic Gaussian (LQG) Control for Output Feedback Systems](https://mlsanity.com/paper/2011.09929/) (2020)
- [Model-Free Linear Quadratic Control via Reduction to Expert Prediction](https://mlsanity.com/paper/1804.06021/) (2018)
- [Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with √T Regret](https://mlsanity.com/paper/2102.12608/) (2021)
- [Regret Lower Bounds for Learning Linear Quadratic Gaussian Systems](https://mlsanity.com/paper/2201.01680/) (2022)
- [Learning to Control Linear Systems can be Hard](https://mlsanity.com/paper/2205.14035/) (2022)
- [On Uninformative Optimal Policies in Adaptive LQR with Unknown B-Matrix](https://mlsanity.com/paper/2011.09288/) (2020)
- [Learning robust control for LQR systems with multiplicative noise via policy gradient](https://mlsanity.com/paper/1905.13547/) (2019)
- [Recovering Robustness in Model-Free Reinforcement learning](https://mlsanity.com/paper/1810.09337/) (2018)
- [Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator](https://mlsanity.com/paper/1910.01249/) (2019)

## Built on by (most cited first, 10 of 10)

- [Statistical Learning Theory for Control: A Finite Sample Perspective](https://mlsanity.com/paper/2209.05423/) (2022)
- [Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data](https://mlsanity.com/paper/2308.04428/) (2023)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Asynchronous Heterogeneous Linear Quadratic Regulator Design](https://mlsanity.com/paper/2404.09061/) (2024)
- [Oracle Complexity Reduction for Model-free LQR: A Stochastic Variance-Reduced Policy Gradient Approach](https://mlsanity.com/paper/2309.10679/) (2023)
- [Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data](https://mlsanity.com/paper/2502.19977/) (2025)
- [On the Hardness of Learning to Stabilize Linear Systems](https://mlsanity.com/paper/2311.11151/) (2023)
- [Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient](https://mlsanity.com/paper/2403.05367/) (2024)
- [MR-ARL: Model Reference Adaptive Reinforcement Learning for Robustly Stable On-Policy Data-Driven LQR](https://mlsanity.com/paper/2402.14483/) (2024)

## Abstract (arXiv)

We study stochastic policy gradient methods from the perspective of control-theoretic limitations. Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates. We also give an example of a class of stable systems in which policy gradient methods suffer from the curse of dimensionality. Our results apply to both state feedback and partially observed systems.