mlsanityData to 14 Oct 2026

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

arXiv 2303.08431 ↗First posted 15 Mar 2023cs.LG math.OC stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Nonlinear control systems with partial information to the decision maker are prevalent in a variety of applications. As a step toward studying such nonlinear systems, this work explores reinforcement learning methods for finding the optimal policy in the nearly linear-quadratic regulator systems. In particular, we consider a dynamic system that combines linear and nonlinear components, and is governed by a policy with the same structure. Assuming that the nonlinear component comprises kernels with small Lipschitz coefficients, we characterize the optimization landscape of the cost function. Although the cost function is nonconvex in general, we establish the local strong convexity and smoothness in the vicinity of the global optimizer. Additionally, we propose an initialization mechanism to leverage these properties. Building on the developments, we design a policy gradient algorithm that is guaranteed to converge to the globally optimal policy with a linear rate.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 23 of its 56 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

23 of 56 references.

References · 23

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  6. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
  7. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  8. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  9. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  10. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  11. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
  12. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
11 more
  1. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  2. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  3. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
  4. 2019LQR through the Lens of First Order Methods: Discrete-time Casecited by 40
  5. 2020Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycited by 34
  6. 2019Feedback Linearization based on Gaussian Processes with event-triggered Online Learningcited by 24
  7. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 21
  8. 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
  9. 2020Convergence Guarantees of Policy Optimization Methods for Markovian Jump Linear Systemscited by 7
  10. 2020On the Analysis of Model-free Methods for the Linear Quadratic Regulatorcited by 6
  11. 2022Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learningcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 1.

5 papers.

  1. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  2. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  3. 2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 13
  4. 2023Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity and Last-Iterate Convergencecited by 4
  5. 2025Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Datacited by 2