mlsanityData to 14 Oct 2026

Achieving High Accuracy with PINNs via Energy Natural Gradients

arXiv 2302.13163 ↗First posted 25 Feb 2023cs.LG cs.NA math.NAcited by 10

Not on the map. Why?

Abstract (arXiv)

We propose energy natural gradient descent, a natural gradient method with respect to a Hessian-induced Riemannian metric as an optimization algorithm for physics-informed neural networks (PINNs) and the deep Ritz method. As a main motivation we show that the update direction in function space resulting from the energy natural gradient corresponds to the Newton direction modulo an orthogonal projection onto the model's tangent space. We demonstrate experimentally that energy natural gradient descent yields highly accurate solutions with errors several orders of magnitude smaller than what is obtained when training PINNs with standard optimizers like gradient descent or Adam, even when those are allowed significantly more computation time.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 25 Feb 2023.

Builds on

30 of 55 references.

References · 30

  1. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  2. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 678
  3. 2021Characterizing possible failure modes in physics-informed neural networkscited by 643
  4. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  5. 2019DeepXDE: A deep learning library for solving differential equationscited by 619
  6. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  7. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  8. 2014New insights and perspectives on the natural gradient methodcited by 443
  9. 2021Neural Operator: Learning Maps Between Function Spacescited by 414
  10. 2022A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networkscited by 236
  11. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 191
  12. 2013Revisiting Natural Gradient for Deep Networkscited by 159
18 more
  1. 2022Respecting causality is all you need for training physics-informed neural networkscited by 156
  2. 2021Efficient training of physics-informed neural networks via importance samplingcited by 110
  3. 2020Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learningcited by 77
  4. 2020An overview on deep learning-based approximation methods for partial differential equationscited by 65
  5. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  6. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 33
  7. 2022Investigating molecular transport in the human brain from MRI with physics-informed neural networkscited by 24
  8. 2018Natural gradient via optimal transportcited by 20
  9. 2020On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNscited by 19
  10. 2022Competitive Physics Informed Networkscited by 19
  11. 2021Wasserstein Proximal of GANscited by 15
  12. 2021Error Estimates for the Deep Ritz Method with Boundary Penaltycited by 14
  13. 2022PSO-PINN: Physics-Informed Neural Networks Trained with Particle Swarm Optimizationcited by 11
  14. 2022Invariance Properties of the Natural Gradient in Overparametrised Systemscited by 11
  15. 2020Sinkhorn Natural Gradient for Generative Modelscited by 10
  16. 2022Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problemscited by 7
  17. 2021Hessian informed mirror descentcited by 6
  18. 2016An iteration count estimate for a mesh-dependent steepest descent method based on finite elements and Riesz inner product representationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 4; 2026: 4.

10 papers.

  1. 2024Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precisioncited by 3
  2. 2023Physics-Informed Deep Learning of Rate-and-State Fault Frictioncited by 2
  3. 2026Natural gradient descent with momentumcited by 2
  4. 2025Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebracited by 1
  5. 2025Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherencecited by 1
  6. 2025Neural network-driven domain decomposition for efficient solutions to the Helmholtz equationcited by 0
  7. 2025Solving PDEs With Deep Neural Nets under General Boundary Conditionscited by 0
  8. 2026Gauss-Newton Natural Gradient Descent for Shape Learningcited by 0
  9. 2026Understanding Sample Efficiency in Predictive Codingcited by 0
  10. 2026SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learningcited by 0