# Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2408.00573 · HTML: https://mlsanity.com/paper/2408.00573/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2024.

## Built on by (most cited first, 5 of 5)

- [ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning](https://mlsanity.com/paper/2412.10782/) (2024)
- [Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation](https://mlsanity.com/paper/2508.21571/) (2025)
- [On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks](https://mlsanity.com/paper/2410.08041/) (2024)
- [Feature Learning for the High Dimensional Stationary Sch"odinger Equation with Deep Ritz Method](https://mlsanity.com/paper/2607.06369/) (2026)
- [The Differential Neural Tangent Kernel and Its Positivity](https://mlsanity.com/paper/2607.10200/) (2026)

## Abstract (arXiv)

In the context of over-parameterization, there is a line of work demonstrating that randomly initialized (stochastic) gradient descent (GD) converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. However, the learning rate of GD for training two-layer neural networks exhibits poor dependence on the sample size and the Gram matrix, leading to a slow training process. In this paper, we show that for training two-layer ReLU3 Physics-Informed Neural Networks (PINNs), the learning rate can be improved from O(λ0) to O(1/|bmH∞|2), implying that GD actually enjoys a faster convergence rate. Despite such improvements, the convergence rate is still tied to the least eigenvalue of the Gram matrix, leading to slow convergence. We then develop the positive definiteness of Gram matrices with general smooth activation functions and provide the convergence analysis of natural gradient descent (NGD) in training two-layer PINNs, demonstrating that the learning rate can be O(1) and at this rate, the convergence rate is independent of the Gram matrix. In particular, for smooth activation functions, the convergence rate of NGD is quadratic. Numerical experiments are conducted to verify our theoretical results.