mlsanityData to 14 Oct 2026

Wasserstein Policy Optimization

arXiv 2505.00663 ↗First posted 1 May 2025cs.LG cs.AIcited by 13

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We introduce Wasserstein Policy Optimization (WPO), an actor-critic algorithm for reinforcement learning in continuous action spaces. WPO can be derived as an approximation to Wasserstein gradient flow over the space of all policies projected into a finite-dimensional parameter space (e.g., the weights of a neural network), leading to a simple and completely general closed-form update. The resulting algorithm combines many properties of deterministic and classic policy gradient methods. Like deterministic policy gradients, it exploits knowledge of the gradient of the action-value function with respect to the action. Like classic policy gradients, it can be applied to stochastic policies with arbitrary distributions over actions -- without using the reparameterization trick. We show results on the DeepMind Control Suite and a magnetic confinement fusion task which compare favorably with state-of-the-art continuous control methods.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 53 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

28 of 53 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  9. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  10. 2018DeepMind Control Suitecited by 956
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  12. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
16 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  3. 2018Maximum a Posteriori Policy Optimisationcited by 328
  4. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  6. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  7. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  8. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  9. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  10. 2020SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergencecited by 64
  11. 2018Policy Optimization as Wasserstein Gradient Flowscited by 34
  12. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  13. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  14. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  15. 2023Wasserstein Quantum Monte Carlo: A Novel Approach for Solving the Quantum Many-Body Schr"odinger Equationcited by 12
  16. 2023Towards practical reinforcement learning for tokamak magnetic controlcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2026Wasserstein Proximal Policy Gradientcited by 4
  2. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 3
  3. 2025Equivalence of stochastic and deterministic policy gradientscited by 2
  4. 2026Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learningcited by 1
  5. 2026A Gradient Flow Perspective on Minimum MMD Estimationcited by 1
  6. 2025Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spacescited by 0
  7. 2026Trust Regions Sell, But Who's Buying? Overlap Geometry as an Alternative Trust Region for Policy Optimizationcited by 0
  8. 2026Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimizationcited by 0
  9. 2026Ratio-Variance Regularized Policy Optimizationcited by 0
  10. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0
  11. 2026Difference of Convex Programming in the Wasserstein Space with Applications to MMD Optimizationcited by 0
  12. 2026Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Controlcited by 0
1 more
  1. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A note on convergence of Wasserstein policy optimization
  2. 2024Compatible Gradient Approximations for Actor-Critic Algorithms
  3. 2025Wasserstein Barycenter Soft Actor-Critic
  4. 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning
  5. 2024Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions
  6. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  7. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  8. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms