mlsanityData to 14 Oct 2026

A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces

arXiv 2310.02951 ↗First posted 4 Oct 2023math.OC cs.LG math.PRcited by 23

Not on the map. Why?

Abstract (arXiv)

We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space. The flow is a continuous-time analogue of a policy mirror descent method. We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy. Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation. To overcome challenges stemming from the lack of the convexity of the objective function and the discontinuity arising from the entropy regulariser, we leverage the performance difference lemma and the duality relationship between the gradient and mirror descent flows. Our analysis provides a theoretical foundation for developing various discrete policy gradient algorithms.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 32 of its 60 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

32 of 60 references.

References · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2019A Theory of Regularized Markov Decision Processescited by 286
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  6. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  7. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  8. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  9. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  11. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  12. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
20 more
  1. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  2. 2020Mirror Descent Policy Optimizationcited by 106
  3. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  4. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  5. 2013The Information Geometry of Mirror Descentcited by 54
  6. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  7. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  8. 2019LQR through the Lens of First Order Methods: Discrete-time Casecited by 40
  9. 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 34
  10. 2022Policy Optimization over General State and Action Spacescited by 23
  11. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  12. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
  13. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  14. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  15. 2021Remarks on input to state stability of perturbed gradient flows, motivated by model-free feedback control learningcited by 8
  16. 2020Policy Gradient using Weak Derivatives for Reinforcement Learningcited by 7
  17. 2021Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Criticcited by 7
  18. 2022Towards a Theoretical Foundation of Policy Optimization for Learning Control Policiescited by 6
  19. 2021Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settingscited by 5
  20. 2023Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulatorscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 10.

23 papers.

Sort:

By area:

  1. 2024Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalitiescited by 13
  2. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 12
  3. 2024Stein transport for Bayesian inferencecited by 11
  4. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradientscited by 5
  5. 2025PPO in the Fisher-Rao geometrycited by 4
  6. 2025Efficient Learning for Entropy-Regularized Markov Decision Processes via Multilevel Monte Carlocited by 3
  7. 2024Mirror Descent-Ascent for mean-field min-max problemscited by 2
  8. 2025Non-convex entropic mean-field optimization via Best Response flowcited by 2
  9. 2026A note on convergence of Wasserstein policy optimizationcited by 2
  10. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processescited by 1
  11. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
  12. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Gamescited by 1
11 more
  1. 2025Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spacescited by 1
  2. 2026Refined Analysis of Entropy-Regularized Actor-Criticcited by 1
  3. 2024Inference of Utilities and Time Preference in Sequential Decision-Makingcited by 0
  4. 2026Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergencecited by 0
  5. 2026On propagation of chaos for the Fisher-Rao gradient flow in entropic mean-field optimizationcited by 0
  6. 2026Global linear convergence of entropy-regularized softmax policy gradient beyond tabular MDPscited by 0
  7. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0
  8. 2026Randomized Optimal Switching Problem and Related Mirror Descent Flowcited by 0
  9. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  10. 2026An analysis of Mirror-Descent Soft Actor-Criticcited by 0
  11. 2026Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problemscited by 0