# Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2407.02827 · HTML: https://mlsanity.com/paper/2407.02827/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks](https://mlsanity.com/paper/1905.13210/) (2019)
- [An Improved Analysis of Training Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/1906.04688/) (2019)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks](https://mlsanity.com/paper/1902.01384/) (2019)
- [Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound](https://mlsanity.com/paper/1906.03593/) (2019)
- [Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression](https://mlsanity.com/paper/1804.02605/) (2018)
- [Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory](https://mlsanity.com/paper/2006.15733/) (2020)
- [Neural networks-based algorithms for stochastic control and PDEs in finance](https://mlsanity.com/paper/2101.08068/) (2021)
- [Solving Electrical Impedance Tomography with Deep Learning](https://mlsanity.com/paper/1906.03944/) (2019)
- [Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks](https://mlsanity.com/paper/2303.01767/) (2023)

## Built on by (most cited first, 3 of 3)

- [Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation](https://mlsanity.com/paper/2508.21571/) (2025)
- [On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks](https://mlsanity.com/paper/2410.08041/) (2024)
- [The Differential Neural Tangent Kernel and Its Positivity](https://mlsanity.com/paper/2607.10200/) (2026)

## Abstract (arXiv)

The optimization algorithms are crucial in training physics-informed neural networks (PINNs), as unsuitable methods may lead to poor solutions. Compared to the common gradient descent (GD) algorithm, implicit gradient descent (IGD) outperforms it in handling certain multi-scale problems. In this paper, we provide convergence analysis for the IGD in training over-parameterized two-layer PINNs. We first derive the training dynamics of IGD in training two-layer PINNs. Then, over-parameterization allows us to prove that the randomly initialized IGD converges to a globally optimal solution at a linear convergence rate. Moreover, due to the distinct training dynamics of IGD compared to GD, the learning rate can be selected independently of the sample size and the least eigenvalue of the Gram matrix. Additionally, the novel approach used in our convergence analysis imposes a milder requirement on the network width. Finally, empirical results validate our theoretical findings.