# Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems

arXiv: https://arxiv.org/abs/2202.06236 · HTML: https://mlsanity.com/paper/2202.06236/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 13 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks](https://mlsanity.com/paper/1905.10961/) (2019)
- [Hutch++: Optimal Stochastic Trace Estimation](https://mlsanity.com/paper/2010.09649/) (2020)
- [Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study](https://mlsanity.com/paper/1708.07827/) (2017)
- [Natural gradient via optimal transport](https://mlsanity.com/paper/1803.07033/) (2018)
- [A fast approach to optimal transport: The back-and-forth method](https://mlsanity.com/paper/1905.12154/) (2019)
- [Wasserstein Proximal of GANs](https://mlsanity.com/paper/2102.06862/) (2021)
- [Optimal transport natural gradient for statistical manifolds with continuous sample space](https://mlsanity.com/paper/1805.08380/) (2018)
- [Kernelized Wasserstein Natural Gradient](https://mlsanity.com/paper/1910.09652/) (2019)
- [Sinkhorn Natural Gradient for Generative Models](https://mlsanity.com/paper/2011.04162/) (2020)
- [Optimal Transport Based Seismic Inversion: Beyond Cycle Skipping](https://mlsanity.com/paper/2002.00031/) (2020)
- [Splitting methods for a class of non-potential mean field games](https://mlsanity.com/paper/2007.00099/) (2020)
- [Solving Large-Scale Optimization Problems with a Convergence Rate Independent of Grid Size](https://mlsanity.com/paper/1805.09453/) (2018)
- [Natural Gradient for Combined Loss Using Wavelets](https://mlsanity.com/paper/2006.15806/) (2020)
- [A Formalization of The Natural Gradient Method for General Similarity Measures](https://mlsanity.com/paper/1902.08959/) (2019)
- [Parametric Fokker-Planck equation](https://mlsanity.com/paper/1903.10076/) (2019)

## Built on by (most cited first, 7 of 7)

- [Taming Hyperparameter Tuning in Continuous Normalizing Flows Using the JKO Scheme](https://mlsanity.com/paper/2211.16757/) (2022)
- [Geometry and convergence of natural policy gradient methods](https://mlsanity.com/paper/2211.02105/) (2022)
- [Achieving High Accuracy with PINNs via Energy Natural Gradients](https://mlsanity.com/paper/2302.13163/) (2023)
- [GAS: A Gaussian Mixture Distribution-Based Adaptive Sampling Method for PINNs](https://mlsanity.com/paper/2303.15849/) (2023)
- [Riemannian Natural Gradient Methods](https://mlsanity.com/paper/2207.07287/) (2022)
- [Conjugate Natural Selection](https://mlsanity.com/paper/2208.13898/) (2022)
- [Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers](https://mlsanity.com/paper/2607.22004/) (2026)

## Abstract (arXiv)

We propose efficient numerical schemes for implementing the natural gradient descent (NGD) for a broad range of metric spaces with applications to PDE-based optimization problems. Our technique represents the natural gradient direction as a solution to a standard least-squares problem. Hence, instead of calculating, storing, or inverting the information matrix directly, we apply efficient methods from numerical linear algebra. We treat both scenarios where the Jacobian, i.e., the derivative of the state variable with respect to the parameter, is either explicitly known or implicitly given through constraints. We can thus reliably compute several natural NGDs for a large-scale parameter space. In particular, we are able to compute Wasserstein NGD in thousands of dimensions, which was believed to be out of reach. Finally, our numerical results shed light on the qualitative differences between the standard gradient descent and various NGD methods based on different metric spaces in nonconvex optimization problems.