mlsanityData to 7 Oct 2026

The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks

arXiv 2402.03864 ↗First posted 6 Feb 2024cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

The Neural Tangent Kernel (NTK) viewpoint is widely employed to analyze the training dynamics of overparameterized Physics-Informed Neural Networks (PINNs). However, unlike the case of linear Partial Differential Equations (PDEs), we show how the NTK perspective falls short in the nonlinear scenario. Specifically, we establish that the NTK yields a random matrix at initialization that is not constant during training, contrary to conventional belief. Another significant difference from the linear regime is that, even in the idealistic infinite-width limit, the Hessian does not vanish and hence it cannot be disregarded during training. This motivates the adoption of second-order optimization methods. We explore the convergence guarantees of such methods in both linear and nonlinear cases, addressing challenges such as spectral bias and slow convergence. Every theoretical result is supported by numerical examples with both linear and nonlinear PDEs, and we highlight the benefits of second-order methods in benchmark test cases.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

21 of 44 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  5. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2020B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Datacited by 329
  8. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  9. 2018Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equationscited by 233
  10. 2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 142
  11. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  12. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
9 more
  1. 2023PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEscited by 84
  2. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  3. 2022Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodologycited by 43
  4. 2021Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domainscited by 37
  5. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  6. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  7. 2022Improved Training of Physics-Informed Neural Networks with Model Ensemblescited by 19
  8. 2021Explicit Convergence Rates of Greedy and Random Quasi-Newton Methodscited by 5
  9. 2023Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orderscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 7.

21 papers.

Sort:

By area:

  1. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  2. 2024Physics-informed kernel learningcited by 11
  3. 2025Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunitiescited by 10
  4. 2025Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networkscited by 6
  5. 2026Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operatorscited by 6
  6. 2025Continuous Representation Methods, Theories, and Applications: An Overview and Perspectivescited by 4
  7. 2025Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operatorscited by 3
  8. 2025Riemann Tensor Neural Networks: Learning Conservative Systems with Physics-Constrained Networkscited by 2
  9. 2025Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equationcited by 2
  10. 2024PINNfluence: Interpreting PINNs through Influence Functionscited by 1
  11. 2025Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebracited by 1
  12. 2025Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulationcited by 1
9 more
  1. 2026Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearitycited by 1
  2. 2026Unveiling the Mechanism of Continuous Representation Full-Waveform Inversion: A Wave Based Neural Tangent Kernel Frameworkcited by 1
  3. 2024On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networkscited by 0
  4. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  5. 2025PINN Balls: Scaling Second-Order Methods for PINNs with Domain Decomposition and Adaptive Samplingcited by 0
  6. 2026Learning the Stellar Structure Equations via Self-supervised Physics-Informed Neural Networkscited by 0
  7. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
  8. 2026Structure-Preserving Neural Surrogates with Tractable Uncertainty Quantificationcited by 0
  9. 2026Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinatescited by 0