Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control
Not on the map. Why?
Abstract (arXiv)
Policy gradient methods, where one searches for the policy of interest by maximizing the value functions using first-order information, become increasingly popular for sequential decision making in reinforcement learning, games, and control. Guaranteeing the global optimality of policy gradient methods, however, is highly nontrivial due to nonconcavity of the value functions. In this exposition, we highlight recent progresses in understanding and developing policy gradient methods with global convergence guarantees, putting an emphasis on their finite-time convergence rates with regard to salient problem parameters.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 8 Oct 2023.
Builds on
24 of 39 references.
References · 24
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
- 2013Optimization, Learning, and Games with Predictable Sequencescited by 188
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
- 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
- 2017Cycles in adversarial regularized learningcited by 124
- 2022On the Convergence Rates of Policy Gradient Methodscited by 119
- 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
- 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 101
- 2018Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimizationcited by 98
12 more
- 2021Near-Optimal No-Regret Learning in General Gamescited by 85
- 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
- 2021Last-iterate Convergence of Decentralized Optimistic Gradient Descent/Ascent in Infinite-horizon Competitive Markov Gamescited by 69
- 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
- 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
- 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
- 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
- 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
- 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
- 2022Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Gamescited by 18
- 2022Block Policy Mirror Descentcited by 11
- 2022Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergencecited by 8
Built on by
Citing papers per year: 2025: 2.
2 papers.