mlsanityData to 7 Oct 2026

Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings

arXiv 2111.00185 ↗First posted 30 Oct 2021cs.LG cs.AI cs.SYcited by 5

Not on the map. Why?

Abstract (arXiv)

Policy gradient methods have been frequently applied to problems in control and reinforcement learning with great success, yet existing convergence analysis still relies on non-intuitive, impractical and often opaque conditions. In particular, existing rates are achieved in limited settings, under strict regularity conditions. In this work, we establish explicit convergence rates of policy gradient methods, extending the convergence regime to weakly smooth policy classes with L2 integrable gradient. We provide intuitive examples to illustrate the insight behind these new conditions. Notably, our analysis also shows that convergence rates are achievable for both the standard policy gradient and the natural policy gradient algorithms under these assumptions. Lastly we provide performance guarantees for the converged policies.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 24 of its 56 references that are in the map. First posted on arXiv 30 Oct 2021.

Builds on

24 of 56 references.

References · 24

  1. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  2. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
  4. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  5. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  6. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  7. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  8. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  9. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  10. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  11. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  12. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
12 more
  1. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  2. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  3. 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
  4. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  5. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 42
  6. 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 27
  7. 2018Near-Optimal Time and Sample Complexities for Solving Discounted Markov Decision Process with a Generative Modelcited by 22
  8. 2017Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselinescited by 20
  9. 2019Smoothing Policies and Safe Policy Gradientscited by 18
  10. 2015Perturbation theory for Markov chains via Wasserstein distancecited by 16
  11. 2019Comments on the Du-Kakade-Wang-Yang Lower Boundscited by 16
  12. 2020Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithmscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 0; 2025: 0; 2026: 1.

5 papers.

  1. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  2. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 19
  3. 2022Geometry and convergence of natural policy gradient methodscited by 12
  4. 2022On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrizationcited by 3
  5. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1