# Achieving High Accuracy with PINNs via Energy Natural Gradients

arXiv: https://arxiv.org/abs/2302.13163 · HTML: https://mlsanity.com/paper/2302.13163/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 25 Feb 2023.

## Builds on (most cited first, 30 of 30)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Efficient training of physics-informed neural networks via importance sampling](https://mlsanity.com/paper/2104.12325/) (2021)
- [Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning](https://mlsanity.com/paper/2008.13333/) (2020)
- [An overview on deep learning-based approximation methods for partial differential equations](https://mlsanity.com/paper/2012.12348/) (2020)
- [Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems](https://mlsanity.com/paper/1905.11675/) (2019)
- [Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks](https://mlsanity.com/paper/1906.02353/) (2019)
- [Investigating molecular transport in the human brain from MRI with physics-informed neural networks](https://mlsanity.com/paper/2205.02592/) (2022)
- [Natural gradient via optimal transport](https://mlsanity.com/paper/1803.07033/) (2018)
- [On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs](https://mlsanity.com/paper/2006.02409/) (2020)
- [Competitive Physics Informed Networks](https://mlsanity.com/paper/2204.11144/) (2022)
- [Wasserstein Proximal of GANs](https://mlsanity.com/paper/2102.06862/) (2021)
- [Error Estimates for the Deep Ritz Method with Boundary Penalty](https://mlsanity.com/paper/2103.01007/) (2021)
- [PSO-PINN: Physics-Informed Neural Networks Trained with Particle Swarm Optimization](https://mlsanity.com/paper/2202.01943/) (2022)
- [Invariance Properties of the Natural Gradient in Overparametrised Systems](https://mlsanity.com/paper/2206.15273/) (2022)
- [Sinkhorn Natural Gradient for Generative Models](https://mlsanity.com/paper/2011.04162/) (2020)
- [Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems](https://mlsanity.com/paper/2202.06236/) (2022)
- [Hessian informed mirror descent](https://mlsanity.com/paper/2106.13477/) (2021)
- [An iteration count estimate for a mesh-dependent steepest descent method based on finite elements and Riesz inner product representation](https://mlsanity.com/paper/1606.08069/) (2016)

## Built on by (most cited first, 10 of 10)

- [Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precision](https://mlsanity.com/paper/2407.17213/) (2024)
- [Physics-Informed Deep Learning of Rate-and-State Fault Friction](https://mlsanity.com/paper/2312.09403/) (2023)
- [Natural gradient descent with momentum](https://mlsanity.com/paper/2604.15554/) (2026)
- [Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2505.11638/) (2025)
- [Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherence](https://mlsanity.com/paper/2509.00024/) (2025)
- [Neural network-driven domain decomposition for efficient solutions to the Helmholtz equation](https://mlsanity.com/paper/2511.15445/) (2025)
- [Solving PDEs With Deep Neural Nets under General Boundary Conditions](https://mlsanity.com/paper/2512.15771/) (2025)
- [Gauss-Newton Natural Gradient Descent for Shape Learning](https://mlsanity.com/paper/2602.00099/) (2026)
- [Understanding Sample Efficiency in Predictive Coding](https://mlsanity.com/paper/2605.11911/) (2026)
- [SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning](https://mlsanity.com/paper/2608.29448/) (2026)

## Abstract (arXiv)

We propose energy natural gradient descent, a natural gradient method with respect to a Hessian-induced Riemannian metric as an optimization algorithm for physics-informed neural networks (PINNs) and the deep Ritz method. As a main motivation we show that the update direction in function space resulting from the energy natural gradient corresponds to the Newton direction modulo an orthogonal projection onto the model's tangent space. We demonstrate experimentally that energy natural gradient descent yields highly accurate solutions with errors several orders of magnitude smaller than what is obtained when training PINNs with standard optimizers like gradient descent or Adam, even when those are allowed significantly more computation time.