# TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision

arXiv: https://arxiv.org/abs/2404.10771 · HTML: https://mlsanity.com/paper/2404.10771/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Quantum Natural Gradient](https://mlsanity.com/paper/1909.02108/) (2019)
- [Long-time integration of parametric evolution equations with physics-informed DeepONets](https://mlsanity.com/paper/2106.05384/) (2021)
- [Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning](https://mlsanity.com/paper/2008.13333/) (2020)
- [Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks](https://mlsanity.com/paper/1905.10961/) (2019)
- [Evolutional Deep Neural Network](https://mlsanity.com/paper/2103.09959/) (2021)
- [Implicit Neural Spatial Representations for Time-dependent PDEs](https://mlsanity.com/paper/2210.00124/) (2022)
- [Gauge Invariant and Anyonic Symmetric Transformer and RNN Quantum States for Quantum Lattice Models](https://mlsanity.com/paper/2101.07243/) (2021)
- [Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks](https://mlsanity.com/paper/2310.04867/) (2023)
- [Simulating 2+1D Lattice Quantum Electrodynamics at Finite Density with Neural Flow Wavefunctions](https://mlsanity.com/paper/2212.06835/) (2022)
- [Spacetime Neural Network for High Dimensional Quantum Dynamics](https://mlsanity.com/paper/2108.02200/) (2021)
- [ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulation](https://mlsanity.com/paper/2304.01996/) (2023)

## Built on by (most cited first, 4 of 4)

- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Metamizer: a versatile neural optimizer for fast and accurate physics simulations](https://mlsanity.com/paper/2410.19746/) (2024)
- [Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold](https://mlsanity.com/paper/2502.07650/) (2025)
- [Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approaches](https://mlsanity.com/paper/2604.16842/) (2026)

## Abstract (arXiv)

Partial differential equations (PDEs) are instrumental for modeling dynamical systems in science and engineering. The advent of neural networks has initiated a significant shift in tackling these complexities though challenges in accuracy persist, especially for initial value problems. In this paper, we introduce the textitTime-Evolving Natural Gradient (TENG), generalizing time-dependent variational principles and optimization-based time integration, leveraging natural gradient optimization to obtain high accuracy in neural-network-based PDE solutions. Our comprehensive development includes algorithms like TENG-Euler and its high-order variants, such as TENG-Heun, tailored for enhanced precision and efficiency. TENG's effectiveness is further validated through its performance, surpassing current leading methods and achieving textitmachine precision in step-by-step optimizations across a spectrum of PDEs, including the heat equation, Allen-Cahn equation, and Burgers' equation.