mlsanityData to 7 Oct 2026

Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime

arXiv 2201.07296 ↗First posted 18 Jan 2022math.OC cs.AI cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We study the global convergence of policy gradient for infinite-horizon, continuous state and action space, and entropy-regularized Markov decision processes (MDPs). We consider a softmax policy with (one-hidden layer) neural network approximation in a mean-field regime. Additional entropic regularization in the associated mean-field probability measure is added, and the corresponding gradient flow is studied in the 2-Wasserstein metric. We show that the objective function is increasing along the gradient flow. Further, we prove that if the regularization in terms of the mean-field measure is sufficient, the gradient flow converges exponentially fast to the unique stationary solution, which is the unique maximizer of the regularized MDP objective. Lastly, we study the sensitivity of the value function along the gradient flow with respect to regularization parameters and the initial condition. Our results rely on the careful analysis of the non-linear Fokker-Planck-Kolmogorov equation and extend the pioneering work of Mei et al. 2020 and Agarwal et al. 2020, which quantify the global convergence rate of policy gradient for entropy-regularized MDPs in the tabular setting.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 19 of its 69 references that are in the map. First posted on arXiv 18 Jan 2022.

Builds on

19 of 69 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2019A Theory of Regularized Markov Decision Processescited by 279
  6. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  7. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  8. 2017A unified view of entropy-regularized Markov decision processescited by 174
  9. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  10. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  11. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  12. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
7 more
  1. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 46
  2. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  3. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
  4. 2019Mean-Field Neural ODEs via Relaxed Optimal Controlcited by 16
  5. 2019Asymptotics of Reinforcement Learning with Neural Networkscited by 10
  6. 2020Gradient Flows for Regularized Stochastic Control Problemscited by 8
  7. 2019Mean-field Langevin System, Optimal Control and Deep Neural Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 3; 2025: 6; 2026: 4.

20 papers.

Sort:

  1. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  2. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 19
  3. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 19
  4. 2023Beyond Exponentially Fast Mixing in Average-Reward Reinforcement Learning via Multi-Level Monte Carlo Actor-Criticcited by 18
  5. 2022Geometry and convergence of natural policy gradient methodscited by 12
  6. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 9
  7. 2022Linear convergence of a policy gradient method for some finite horizon continuous time control problemscited by 4
  8. 2024Linear convergence of proximal descent schemes on the Wasserstein spacecited by 3
  9. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  10. 2023Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimalitycited by 2
  11. 2025Non-convex entropic mean-field optimization via Best Response flowcited by 2
  12. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
8 more
  1. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
  2. 2025Efficient Computation of Blackwell Optimal Policies using Rational Functionscited by 1
  3. 2026A note on convergence of Wasserstein policy optimizationcited by 1
  4. 2025Meta-reinforcement learning with minimum attentioncited by 0
  5. 2025Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environmentscited by 0
  6. 2026On propagation of chaos for the Fisher-Rao gradient flow in entropic mean-field optimizationcited by 0
  7. 2026Global linear convergence of entropy-regularized softmax policy gradient beyond tabular MDPscited by 0
  8. 2026An analysis of Mirror-Descent Soft Actor-Criticcited by 0