mlsanityData to 7 Oct 2026

Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients

arXiv 2403.19448 ↗First posted 28 Mar 2024math.OC cs.LG cs.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

Kakade's natural policy gradient method has been studied extensively in recent years, showing linear convergence with and without regularization. We study another natural gradient method based on the Fisher information matrix of the state-action distributions which has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

32 of 65 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2021Convex optimizationcited by 2,479
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2018Computational Optimal Transportcited by 765
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  8. 2017A unified view of entropy-regularized Markov decision processescited by 174
  9. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  11. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  12. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
20 more
  1. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  2. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
  3. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  4. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 51
  5. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  6. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  7. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  8. 2018Hessian Riemannian gradient flows in convex programmingcited by 25
  9. 2018An explicit analysis of the entropic penalty in linear programmingcited by 23
  10. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 19
  11. 2022Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrizationcited by 17
  12. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 15
  13. 2021Actor-critic is implicitly biased towards high entropy optimal policiescited by 13
  14. 2022Geometry and convergence of natural policy gradient methodscited by 12
  15. 2022Invariance Properties of the Natural Gradient in Overparametrised Systemscited by 11
  16. 2021The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPscited by 9
  17. 2021Hessian informed mirror descentcited by 6
  18. 2024A Geometric Embedding Approach to Multiple Games and Multiple Populationscited by 6
  19. 2021Approximate Newton policy gradient algorithmscited by 3
  20. 2022Toric Geometry of Entropic Regularizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  2. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9
  3. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processescited by 1
  4. 2025Entropic Mirror Descent for Linear Systems: Polyak's Stepsize and Implicit Biascited by 0
  5. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0