Accelerating Primal-dual Methods for Regularized Markov Decision Processes
Not on the map. Why?
Abstract (arXiv)
Entropy regularized Markov decision processes have been widely used in reinforcement learning. This paper is concerned with the primal-dual formulation of the entropy regularized problems. Standard first-order methods suffer from slow convergence due to the lack of strict convexity and concavity. To address this issue, we first introduce a new quadratically convexified primal-dual formulation. The natural gradient ascent descent of the new formulation enjoys global convergence guarantee and exponential convergence rate. We also propose a new interpolating metric that further accelerates the convergence significantly. Numerical results are provided to demonstrate the performance of the proposed methods under multiple settings.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 21 Feb 2022.
Builds on
22 of 42 references.
References · 22
- 2015Trust Region Policy Optimizationcited by 2,503
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2016OpenAI Gymcited by 1,684
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2019A Theory of Regularized Markov Decision Processescited by 279
- 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
- 2017A unified view of entropy-regularized Markov decision processescited by 174
- 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
- 2018Understanding the impact of entropy on policy optimizationcited by 171
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
10 more
- 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
- 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
- 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 46
- 2017Primal-Dual π Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problemscited by 32
- 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 29
- 2020Cautious Reinforcement Learning via Distributional Risk in the Dual Domaincited by 18
- 2019Faster saddle-point optimization for solving large-scale Markov decision processescited by 8
- 2018Stochastic Primal-Dual Q-Learningcited by 7
- 2020A Note on Optimization Formulations of Markov Decision Processescited by 7
- 2017Deep Primal-Dual Reinforcement Learning: Accelerating Actor-Critic using Bellman Dualitycited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.