mlsanityData to 14 Oct 2026

How are policy gradient methods affected by the limits of control?

arXiv 2206.06863 ↗First posted 14 Jun 2022math.OC cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We study stochastic policy gradient methods from the perspective of control-theoretic limitations. Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates. We also give an example of a class of stable systems in which policy gradient methods suffer from the curse of dimensionality. Our results apply to both state feedback and partially observed systems.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

21 of 33 references.

References · 21

  1. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  2. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
  3. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  4. 2020Naive Exploration is Optimal for Online LQRcited by 116
  5. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
  6. 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 40
  7. 2021On the Sample Complexity of Stability Constrained Imitation Learningcited by 38
  8. 2021Analysis of the Optimization Landscape of Linear Quadratic Gaussian (LQG) Controlcited by 28
  9. 2022Single Trajectory Nonparametric Learning of Nonlinear Dynamicscited by 26
  10. 2020Regret Bounds for Adaptive Nonlinear Controlcited by 25
  11. 2021Stabilizing Dynamical Systems via Policy Gradient Methodscited by 25
  12. 2021Linear Systems can be Hard to Learncited by 20
9 more
  1. 2020Sample Complexity of Linear Quadratic Gaussian (LQG) Control for Output Feedback Systemscited by 16
  2. 2018Model-Free Linear Quadratic Control via Reduction to Expert Predictioncited by 14
  3. 2021Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with √T Regretcited by 12
  4. 2022Regret Lower Bounds for Learning Linear Quadratic Gaussian Systemscited by 11
  5. 2022Learning to Control Linear Systems can be Hardcited by 11
  6. 2020On Uninformative Optimal Policies in Adaptive LQR with Unknown B-Matrixcited by 7
  7. 2019Learning robust control for LQR systems with multiplicative noise via policy gradientcited by 6
  8. 2018Recovering Robustness in Model-Free Reinforcement learningcited by 5
  9. 2019Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulatorcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 3; 2025: 1.

10 papers.

  1. 2022Statistical Learning Theory for Control: A Finite Sample Perspectivecited by 53
  2. 2023Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Datacited by 13
  3. 2022Towards a Theoretical Foundation of Policy Optimization for Learning Control Policiescited by 6
  4. 2023Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithmscited by 6
  5. 2024Asynchronous Heterogeneous Linear Quadratic Regulator Designcited by 4
  6. 2023Oracle Complexity Reduction for Model-free LQR: A Stochastic Variance-Reduced Policy Gradient Approachcited by 2
  7. 2025Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Datacited by 2
  8. 2023On the Hardness of Learning to Stabilize Linear Systemscited by 1
  9. 2024Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradientcited by 1
  10. 2024MR-ARL: Model Reference Adaptive Reinforcement Learning for Robustly Stable On-Policy Data-Driven LQRcited by 0