mlsanityData to 14 Oct 2026

Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence

arXiv 2105.11066 ↗First posted 24 May 2021cs.LG cs.IT math.ITcited by 84

Not on the map. Why?

Cited by 27 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Policy optimization, which finds the desired policy by maximizing value functions via optimization techniques, lies at the heart of reinforcement learning (RL). In addition to value maximization, other practical considerations arise as well, including the need of encouraging exploration, and that of ensuring certain structural properties of the learned policy due to safety, resource and operational constraints. These can often be accounted for via regularized RL, which augments the target value function with a structure-promoting regularizer. Focusing on discounted infinite-horizon Markov decision processes, we propose a generalized policy mirror descent (GPMD) algorithm for solving regularized RL. As a generalization of policy mirror descent (arXiv:2102.00135), our algorithm accommodates a general class of convex regularizers and promotes the use of Bregman divergence in cognizant of the regularizer in use. We demonstrate that our algorithm converges linearly to the global solution over an entire range of learning rates, in a dimension-free fashion, even when the regularizer lacks strong convexity and smoothness. In addition, this linear convergence feature is provably stable in the face of inexact policy evaluation and imperfect policy updates. Numerical experiments are provided to corroborate the appealing performance of GPMD.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Builds on 35 of its 68 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

35 of 68 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  6. 2019A Theory of Regularized Markov Decision Processescited by 286
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  8. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  9. 2017A unified view of entropy-regularized Markov decision processescited by 182
  10. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  11. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  12. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
23 more
  1. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  2. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  3. 2020Exploration-Exploitation in Constrained MDPscited by 149
  4. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  5. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  6. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  7. 2020Mirror Descent Policy Optimizationcited by 106
  8. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  9. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  10. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  11. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  12. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  13. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  14. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
  15. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  16. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 37
  17. 2015An optimal randomized incremental gradient methodcited by 33
  18. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  19. 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
  20. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 19
  21. 2021Provably Efficient Policy Optimization for Two-Player Zero-Sum Markov Gamescited by 18
  22. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  23. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 13; 2024: 21; 2025: 11; 2026: 17.

84 papers.

Sort:

By area:

  1. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  2. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
  3. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 57
  4. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  5. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  6. 2022Stochastic first-order methods for average-reward Markov decision processescited by 32
  7. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  8. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  9. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
  10. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
  11. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
  12. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 25
38 more
  1. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 23
  2. 2022Policy Optimization over General State and Action Spacescited by 23
  3. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  4. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 20
  5. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  6. 2023Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processescited by 18
  7. 2021Bregman Gradient Policy Optimizationcited by 17
  8. 2022Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrizationcited by 17
  9. 2023Improved Regret for Efficient Online Reinforcement Learning with Linear Function Approximationcited by 17
  10. 2022Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexitycited by 14
  11. 2023Policy Mirror Descent Inherently Explores Action Spacecited by 13
  12. 2022Geometry and convergence of natural policy gradient methodscited by 12
  13. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 12
  14. 2024Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergencecited by 11
  15. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 11
  16. 2022Independent Natural Policy Gradient Methods for Potential Games: Finite-time Global Convergence with Entropy Regularizationcited by 10
  17. 2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
  18. 2023When is Agnostic Reinforcement Learning Statistically Tractable?cited by 9
  19. 2022Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergencecited by 8
  20. 2022Robust Imitation via Mirror Descent Inverse Reinforcement Learningcited by 6
  21. 2023On Representation Complexity of Model-based and Model-free Reinforcement Learningcited by 6
  22. 2024Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reductioncited by 6
  23. 2024Independent Policy Mirror Descent for Markov Potential Games: Scaling to Large Number of Playerscited by 6
  24. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rankcited by 6
  25. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradientscited by 5
  26. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  27. 2024GFlowNet Training by Policy Gradientscited by 5
  28. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  29. 2022Algorithm for Constrained Markov Decision Process with Linear Convergencecited by 4
  30. 2025PPO in the Fisher-Rao geometrycited by 4
  31. 2021Approximate Newton policy gradient algorithmscited by 3
  32. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  33. 2025Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Gamescited by 3
  34. 2025On the Convergence of Policy Mirror Descent with Temporal Difference Evaluationcited by 3
  35. 2022Accelerating Primal-dual Methods for Regularized Markov Decision Processescited by 2
  36. 2022Mirror descent of Hopfield modelcited by 2
  37. 2023Fast Computation of Optimal Transport via Entropy-Regularized Extragradient Methodscited by 2
  38. 2023On the Linear Convergence of Policy Gradient under Hadamard Parameterizationcited by 2

50 most cited of 84 All 84 (JSON)