# ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning

arXiv: https://arxiv.org/abs/2412.10782 · HTML: https://mlsanity.com/paper/2412.10782/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 61 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Builds on (most cited first, 25 of 25)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [A Kronecker-factored approximate Fisher matrix for convolution layers](https://mlsanity.com/paper/1602.01407/) (2016)
- [Deep Learning of Vortex Induced Vibrations](https://mlsanity.com/paper/1808.08952/) (2018)
- [Can Physics-Informed Neural Networks beat the Finite Element Method?](https://mlsanity.com/paper/2302.04107/) (2023)
- [Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems](https://mlsanity.com/paper/1905.11675/) (2019)
- [Deep learning of free boundary and Stefan problems](https://mlsanity.com/paper/2006.05311/) (2020)
- [Riemannian metrics for neural networks I: feedforward networks](https://mlsanity.com/paper/1303.0818/) (2013)
- [Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics](https://mlsanity.com/paper/2402.10680/) (2024)
- [An operator preconditioning perspective on training in physics-informed machine learning](https://mlsanity.com/paper/2310.05801/) (2023)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Position: Optimization in SciML Should Employ the Function Space Geometry](https://mlsanity.com/paper/2402.07318/) (2024)
- [Competitive Physics Informed Networks](https://mlsanity.com/paper/2204.11144/) (2022)
- [Deep Learning of Turbulent Scalar Mixing](https://mlsanity.com/paper/1811.07095/) (2018)
- [Preconditioning for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.00531/) (2024)
- [Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally](https://mlsanity.com/paper/2405.19816/) (2024)
- [True Asymptotic Natural Gradient Optimization](https://mlsanity.com/paper/1712.08449/) (2017)
- [Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks](https://mlsanity.com/paper/2408.00573/) (2024)
- [Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradient](https://mlsanity.com/paper/2202.06232/) (2022)

## Built on by (most cited first, 7 of 7)

- [Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.21404/) (2025)
- [Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operators](https://mlsanity.com/paper/2601.14517/) (2026)
- [A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms](https://mlsanity.com/paper/2508.21022/) (2025)
- [Natural gradient descent with momentum](https://mlsanity.com/paper/2604.15554/) (2026)
- [Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks](https://mlsanity.com/paper/2511.04494/) (2025)
- [SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning](https://mlsanity.com/paper/2608.29448/) (2026)
- [Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers](https://mlsanity.com/paper/2609.20641/) (2026)

## Abstract (arXiv)

In the recent years, Physics Informed Neural Networks (PINNs) have received strong interest as a method to solve PDE driven systems, in particular for data assimilation purpose. This method is still in its infancy, with many shortcomings and failures that remain not properly understood. In this paper we propose a natural gradient approach to PINNs which contributes to speed-up and improve the accuracy of the training. Based on an in depth analysis of the differential geometric structures of the problem, we come up with two distinct contributions: (i) a new natural gradient algorithm that scales as min(P2S, S2P), where P is the number of parameters, and S the batch size; (ii) a mathematically principled reformulation of the PINNs problem that allows the extension of natural gradient to it, with proved connections to Green's function theory.