# Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers

arXiv: https://arxiv.org/abs/2410.06308 · HTML: https://mlsanity.com/paper/2410.06308/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations](https://mlsanity.com/paper/2107.07871/) (2021)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Parallel Physics-Informed Neural Networks via Domain Decomposition](https://mlsanity.com/paper/2104.10013/) (2021)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Multi-stage Neural Networks: Function Approximator of Machine Precision](https://mlsanity.com/paper/2307.08934/) (2023)
- [A novel meta-learning initialization method for physics-informed neural networks](https://mlsanity.com/paper/2107.10991/) (2021)
- [Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives](https://mlsanity.com/paper/2305.08466/) (2023)
- [Stacked networks improve physics-informed training: applications to neural networks and deep operator networks](https://mlsanity.com/paper/2311.06483/) (2023)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [NSGA-PINN: A Multi-Objective Optimization Method for Physics-Informed Neural Network Training](https://mlsanity.com/paper/2303.02219/) (2023)
- [DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splitting](https://mlsanity.com/paper/2212.05571/) (2022)
- [Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations](https://mlsanity.com/paper/2405.14099/) (2024)
- [Initialization-enhanced Physics-Informed Neural Network with Domain Decomposition (IDPINN)](https://mlsanity.com/paper/2406.03172/) (2024)
- [HomPINNs: homotopy physics-informed neural networks for solving the inverse problems of nonlinear differential equations with multiple solutions](https://mlsanity.com/paper/2304.02811/) (2023)
- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)

## Built on by (most cited first, 4 of 4)

- [Complex Physics-Informed Neural Network](https://mlsanity.com/paper/2502.04917/) (2025)
- [Learn Singularly Perturbed Solutions via Homotopy Dynamics](https://mlsanity.com/paper/2502.00488/) (2025)
- [Prescribed-Basis Coefficient-to-Coefficient Neural Operator for Partial Differential Equations](https://mlsanity.com/paper/2510.10350/) (2025)
- [Taming the Loss Landscape of PINNs with Noisy Feynman-Kac Supervision: Operator Preconditioning and Non-Asymptotic Error Bounds](https://mlsanity.com/paper/2606.00643/) (2026)

## Abstract (arXiv)

Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods leverage neural networks as basis functions to approximate PDE solutions. However, training such networks can be challenging, often resulting in limited accuracy. In this paper, we investigate the training dynamics of neural network-based PDE solvers with a focus on the impact of initialization techniques. We assess training difficulty by analyzing the eigenvalue distribution of the kernel and apply the concept of effective rank to quantify this difficulty, where a larger effective rank correlates with faster convergence of the training error. Building upon this, we discover through theoretical analysis and numerical experiments that two initialization techniques, partition of unity (PoU) and variance scaling (VS), enhance the effective rank, thereby accelerating the convergence of training error. Furthermore, comprehensive experiments using popular PDE-solving frameworks, such as PINN, Deep Ritz, and the operator learning framework DeepOnet, confirm that these initialization techniques consistently speed up convergence, in line with our theoretical findings.