# Efficient Wasserstein Natural Gradients for Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.05380 · HTML: https://mlsanity.com/paper/2010.05380/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [Confronting Reward Model Overoptimization with Constrained RLHF](https://mlsanity.com/paper/2310.04373/) (2023)
- [Tactical Optimism and Pessimism for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.03765/) (2021)
- [MICo: Improved representations via sampling-based state similarity for Markov decision processes](https://mlsanity.com/paper/2106.08229/) (2021)
- [ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs](https://mlsanity.com/paper/2302.01275/) (2023)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [A First-Occupancy Representation for Reinforcement Learning](https://mlsanity.com/paper/2109.13863/) (2021)
- [Wasserstein Policy Optimization](https://mlsanity.com/paper/2505.00663/) (2025)
- [Learning General World Models in a Handful of Reward-Free Deployments](https://mlsanity.com/paper/2210.12719/) (2022)
- [Geometry and convergence of natural policy gradient methods](https://mlsanity.com/paper/2211.02105/) (2022)
- [Towards an Understanding of Default Policies in Multitask Policy Optimization](https://mlsanity.com/paper/2111.02994/) (2021)
- [Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actions](https://mlsanity.com/paper/2210.11137/) (2022)
- [Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policies](https://mlsanity.com/paper/2305.10411/) (2023)
- [Provably Convergent Policy Optimization via Metric-aware Trust Region Methods](https://mlsanity.com/paper/2306.14133/) (2023)
- [Wasserstein Proximal Policy Gradient](https://mlsanity.com/paper/2603.02576/) (2026)
- [A State Representation for Diminishing Rewards](https://mlsanity.com/paper/2309.03710/) (2023)
- [A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2312.15474/) (2023)
- [Constrained Sliced Wasserstein Embedding](https://mlsanity.com/paper/2506.02203/) (2025)
- [Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment](https://mlsanity.com/paper/2602.01685/) (2026)
- [Minimum Description Length Control](https://mlsanity.com/paper/2207.08258/) (2022)
- [Transfer RL via the Undo Maps Formalism](https://mlsanity.com/paper/2211.14469/) (2022)
- [Wasserstein Diversity-Enriched Regularizer for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2308.00989/) (2023)
- [Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control](https://mlsanity.com/paper/2608.07433/) (2026)
- [Multi-step Proximal Policy Improvement in Offline Reinforcement Learning](https://mlsanity.com/paper/2609.03842/) (2026)

## Abstract (arXiv)

A novel optimization approach is proposed for application to policy gradient methods and evolution strategies for reinforcement learning (RL). The procedure uses a computationally efficient Wasserstein natural gradient (WNG) descent that takes advantage of the geometry induced by a Wasserstein penalty to speed optimization. This method follows the recent theme in RL of including a divergence penalty in the objective to establish a trust region. Experiments on challenging tasks demonstrate improvements in both computational cost and performance over advanced baselines.