# Natural gradient via optimal transport

arXiv: https://arxiv.org/abs/1803.07033 · HTML: https://mlsanity.com/paper/1803.07033/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2018.

## Built on by (most cited first, 20 of 20)

- [Sinkhorn Distributionally Robust Optimization](https://mlsanity.com/paper/2109.11926/) (2021)
- [Wasserstein Proximal of GANs](https://mlsanity.com/paper/2102.06862/) (2021)
- [Geometry and convergence of natural policy gradient methods](https://mlsanity.com/paper/2211.02105/) (2022)
- [Invariance Properties of the Natural Gradient in Overparametrised Systems](https://mlsanity.com/paper/2206.15273/) (2022)
- [Achieving High Accuracy with PINNs via Energy Natural Gradients](https://mlsanity.com/paper/2302.13163/) (2023)
- [Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems](https://mlsanity.com/paper/2202.06236/) (2022)
- [Generative Assignment Flows for Representing and Learning Joint Distributions of Discrete Data](https://mlsanity.com/paper/2406.04527/) (2024)
- [Depth Without the Magic: Inductive Bias of Natural Gradient Descent](https://mlsanity.com/paper/2111.11542/) (2021)
- [Transport information Bregman divergences](https://mlsanity.com/paper/2101.01162/) (2021)
- [Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradient](https://mlsanity.com/paper/2202.06232/) (2022)
- [Gradient flow in parameter space is equivalent to linear interpolation in output space](https://mlsanity.com/paper/2408.01517/) (2024)
- [Sparse Variational Student-t Processes for Heavy-tailed Modeling](https://mlsanity.com/paper/2408.06699/) (2024)
- [Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms](https://mlsanity.com/paper/2410.19055/) (2024)
- [A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations](https://mlsanity.com/paper/2411.06278/) (2024)
- [Manifold learning in metric spaces](https://mlsanity.com/paper/2503.16187/) (2025)
- [ENIGMA: The Geometry of Reasoning and Alignment in Large-Language Models](https://mlsanity.com/paper/2510.11278/) (2025)
- [A Transport-Based Geometry of Belief-Cost](https://mlsanity.com/paper/2606.21585/) (2026)
- [Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers](https://mlsanity.com/paper/2607.22004/) (2026)
- [The Dually Flat Geometry of Planning as Inference](https://mlsanity.com/paper/2609.04005/) (2026)
- [Discrete Action Matching: Learning Stochastic Dynamics from Samples via State Graphs](https://mlsanity.com/paper/2610.05071/) (2026)

## Abstract (arXiv)

We study a natural Wasserstein gradient flow on manifolds of probability distributions with discrete sample spaces. We derive the Riemannian structure for the probability simplex from the dynamical formulation of the Wasserstein distance on a weighted graph. We pull back the geometric structure to the parameter space of any given probability model, which allows us to define a natural gradient flow there. In contrast to the natural Fisher-Rao gradient, the natural Wasserstein gradient incorporates a ground metric on sample space. We illustrate the analysis of elementary exponential family examples and demonstrate an application of the Wasserstein natural gradient to maximum likelihood estimation.