mlsanityData to 7 Oct 2026

Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

arXiv 2106.04096 ↗First posted 8 Jun 2021cs.LG math.OC stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Natural policy gradient (NPG) methods with entropy regularization achieve impressive empirical success in reinforcement learning problems with large state-action spaces. However, their convergence properties and the impact of entropy regularization remain elusive in the function approximation regime. In this paper, we establish finite-time convergence analyses of entropy-regularized NPG with linear function approximation under softmax parameterization. In particular, we prove that entropy-regularized NPG with averaging satisfies the persistence of excitation condition, and achieves a fast convergence rate of tildeO(1/T) up to a function approximation error in regularized Markov decision processes. This convergence result does not require any a priori assumptions on the policies. Furthermore, under mild regularity conditions on the concentrability coefficient and basis vectors, we prove that entropy-regularized NPG exhibits linear convergence up to a function approximation error.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

25 of 55 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2014New insights and perspectives on the natural gradient methodcited by 435
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  9. 2019A Theory of Regularized Markov Decision Processescited by 279
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  11. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  12. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
13 more
  1. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  2. 2017A unified view of entropy-regularized Markov decision processescited by 174
  3. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  4. 2018Understanding the impact of entropy on policy optimizationcited by 171
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  6. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  7. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  8. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  9. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  10. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  11. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 42
  12. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  13. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 4; 2025: 5; 2026: 5.

15 papers.

Sort:

By area:

  1. 2022Policy Optimization over General State and Action Spacescited by 21
  2. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  3. 2024Recurrent Natural Policy Gradient for POMDPscited by 8
  4. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradientscited by 5
  5. 2025Convergence of Policy Mirror Descent Beyond Compatible Function Approximationcited by 2
  6. 2025Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Datacited by 2
  7. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processescited by 1
  8. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
  9. 2025Sampling Complexity of TD and PPO in RKHScited by 1
  10. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 1
  11. 2025Digital Twin Calibration with Model-Based Reinforcement Learningcited by 0
  12. 2026Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Frameworkcited by 0
3 more
  1. 2026Global linear convergence of entropy-regularized softmax policy gradient beyond tabular MDPscited by 0
  2. 2026An analysis of Mirror-Descent Soft Actor-Criticcited by 0
  3. 2026FERPO: Forward Entropy-Regularized Policy Optimizationcited by 0