mlsanityData to 14 Oct 2026

A note on convergence of Wasserstein policy optimization

arXiv 2605.22622 ↗First posted 21 May 2026cs.LG math.OCcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Wasserstein Policy Optimization (WPO) is a recently proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize stochastic policies in continuous action spaces. Despite its empirical success, the theoretical convergence properties of WPO in environments with continuous state and action spaces have yet to be fully established. In this note, we argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly. This is done by leveraging recent advances in mean-field analysis for convergence of gradient flows using log-Sobole inequalities. Assuming existence of sufficiently regular solution to the gradient flow equation we demonstrate monotonic energy dissipation along the flow and establish a local log-Sobolev inequality. Ultimately, these properties allow us to argue that the value function should converge linearly to the global optimum.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 10 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

8 of 10 references.

References · 8

  1. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  3. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 61
  4. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 55
  5. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  6. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  7. 2021Convergence Rates of Gradient Methods for Convex Optimization in the Space of Measurescited by 20
  8. 2024Linear convergence of proximal descent schemes on the Wasserstein spacecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learningcited by 1
  2. 2026Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Wasserstein Proximal Policy Gradient
  2. 2025Wasserstein Policy Optimization
  3. 2026Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
  4. 2026Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity
  5. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
  6. 2025Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces
  7. 2025Convergence and Sample Complexity of First-Order Methods for Agnostic Reinforcement Learning
  8. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints