# Sinkhorn Natural Gradient for Generative Models

arXiv: https://arxiv.org/abs/2011.04162 · HTML: https://mlsanity.com/paper/2011.04162/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 10 of 10)

- [FlashSinkhorn: IO-Aware Entropic Optimal Transport on GPU](https://mlsanity.com/paper/2602.03067/) (2026)
- [Achieving High Accuracy with PINNs via Energy Natural Gradients](https://mlsanity.com/paper/2302.13163/) (2023)
- [Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems](https://mlsanity.com/paper/2202.06236/) (2022)
- [Stochastic Modified Flows for Riemannian Stochastic Gradient Descent](https://mlsanity.com/paper/2402.03467/) (2024)
- [Sparse Variational Student-t Processes for Heavy-tailed Modeling](https://mlsanity.com/paper/2408.06699/) (2024)
- [A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations](https://mlsanity.com/paper/2411.06278/) (2024)
- [On a Relation Between the Rate-Distortion Function and Optimal Transport](https://mlsanity.com/paper/2307.00246/) (2023)
- [Neural Hamilton--Jacobi Characteristic Flows for Optimal Transport](https://mlsanity.com/paper/2510.01153/) (2025)
- [Implicit Neural Optimal Transport via Fixed-Point Optimization](https://mlsanity.com/paper/2605.10792/) (2026)
- [Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers](https://mlsanity.com/paper/2607.22004/) (2026)

## Abstract (arXiv)

We consider the problem of minimizing a functional over a parametric family of probability measures, where the parameterization is characterized via a push-forward structure. An important application of this problem is in training generative adversarial networks. In this regard, we propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence. We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically with respect to the desired accuracy. This is in sharp contrast to existing natural gradient methods that can only be carried out approximately. Moreover, in practical applications when only Monte-Carlo type integration is available, we design an empirical estimator for SIM and provide the stability analysis. In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.