mlsanityData to 14 Oct 2026

On the Linear Convergence of Policy Gradient under Hadamard Parameterization

arXiv 2305.19575 ↗First posted 31 May 2023math.OC cs.LG stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

The convergence of deterministic policy gradient under the Hadamard parameterization is studied in the tabular setting and the linear convergence of the algorithm is established. To this end, we first show that the error decreases at an O(frac1k) rate for all the iterations. Based on this result, we further show that the algorithm has a faster local linear convergence rate after k0 iterations, where k0 is a constant that only depends on the MDP problem and the initialization. To show the local linear convergence of the algorithm, we have indeed established the contraction of the sub-optimal probability bsk (i.e., the probability of the output policy πk on non-optimal actions) when k≥ k0.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

25 of 44 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  4. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  5. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  6. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  7. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  8. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  9. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  10. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  11. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  12. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
13 more
  1. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  2. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  3. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  4. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  5. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  6. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  7. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  8. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  9. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  10. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  11. 2023Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processescited by 18
  12. 2022Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexitycited by 14
  13. 2021From the simplex to the sphere: Faster constrained optimization using the Hadamard parametrizationcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimizationcited by 17
  2. 2025Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Datacited by 2