Wasserstein Proximal Policy Gradient
Not on the map. Why?
Abstract (arXiv)
We study policy gradient methods for continuous-action, entropy-regularized reinforcement learning through the lens of Wasserstein geometry. Starting from a Wasserstein proximal update, we derive Wasserstein Proximal Policy Gradient (WPPG) via an operator-splitting scheme that alternates an optimal transport update with a heat step implemented by Gaussian convolution. This formulation avoids evaluating the policy's log density or its gradient, making the method directly applicable to expressive implicit stochastic policies specified as pushforward maps. We establish a global linear convergence rate for WPPG, covering both exact policy evaluation and actor-critic implementations with controlled approximation error. Empirically, WPPG is simple to implement and attains competitive performance on standard continuous-control benchmarks.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 35 references that are in the map. First posted on arXiv 3 Mar 2026.
Builds on
17 of 35 references.
References · 17
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Trust Region Policy Optimizationcited by 2,503
- 2015End-to-End Training of Deep Visuomotor Policiescited by 847
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
- 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
- 2022On the Convergence Rates of Policy Gradient Methodscited by 115
- 2019Wasserstein Robust Reinforcement Learningcited by 50
- 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
- 2018Policy Optimization as Wasserstein Gradient Flowscited by 33
5 more
- 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
- 2025Wasserstein Policy Optimizationcited by 12
- 2022Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actionscited by 8
- 2018Implicit Policy for Reinforcement Learningcited by 6
- 2023Provably Convergent Policy Optimization via Metric-aware Trust Region Methodscited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.