mlsanityData to 14 Oct 2026

An operator preconditioning perspective on training in physics-informed machine learning

arXiv 2310.05801 ↗First posted 9 Oct 2023cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the behavior of gradient descent algorithms in physics-informed machine learning methods like PINNs, which minimize residuals connected to partial differential equations (PDEs). Our key result is that the difficulty in training these models is closely related to the conditioning of a specific differential operator. This operator, in turn, is associated to the Hermitian square of the differential operator of the underlying PDE. If this operator is ill-conditioned, it results in slow or infeasible training. Therefore, preconditioning this operator is crucial. We employ both rigorous mathematical analysis and empirical evaluations to investigate various strategies, explaining how they better condition this critical operator, and consequently improve training.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 23 of its 50 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

23 of 50 references.

References · 23

  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  2. 2018On the Spectral Bias of Neural Networkscited by 1,341
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2021Characterizing possible failure modes in physics-informed neural networkscited by 643
  5. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  6. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
  7. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  8. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 540
  9. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 527
  10. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  11. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 214
  12. 2017Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equationscited by 208
11 more
  1. 2022Respecting causality is all you need for training physics-informed neural networkscited by 156
  2. 2019Variational Physics-Informed Neural Networks For Solving Partial Differential Equationscited by 148
  3. 2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 145
  4. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  5. 2020Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEscited by 116
  6. 2021Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEscited by 57
  7. 2022Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equationscited by 56
  8. 2020Physics Informed Neural Networks for Simulating Radiative Transfercited by 35
  9. 2022wPINNs: Weak Physics informed neural networks for approximating entropy solutions of hyperbolic conservation lawscited by 29
  10. 2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
  11. 2023Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equationscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 8; 2026: 7.

27 papers.

Sort:

  1. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 113
  2. 2024Numerical analysis of physics-informed neural networks and related models in physics-informed machine learningcited by 39
  3. 2024Parameterized Physics-informed Neural Networks for Parameterized PDEscited by 38
  4. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  5. 2024Dynamical Measure Transport and Neural PDE Solvers for Samplingcited by 20
  6. 2024Geometry-Informed Neural Networkscited by 12
  7. 2024PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representationscited by 11
  8. 2025Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEscited by 11
  9. 2023Transfer learning for improved generalizability in causal physics-informed neural networks for beam simulationscited by 8
  10. 2025Enhancing Physics-Informed Neural Networks Through Feature Engineeringcited by 8
  11. 2024ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learningcited by 7
  12. 2025Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networkscited by 6
15 more
  1. 2024Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applicationscited by 5
  2. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5
  3. 2024A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networkscited by 2
  4. 2025Beyond Accuracy: EcoL2 Metric for Sustainable Neural PDE Solverscited by 2
  5. 2025Leveraging Influence Functions for Resampling Data in Physics-Informed Neural Networkscited by 2
  6. 2024PINNfluence: Interpreting PINNs through Influence Functionscited by 1
  7. 2025Unveiling the Potential of Superexpressive Networks in Implicit Neural Representationscited by 1
  8. 2025Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebracited by 1
  9. 2026Gauss-Newton Natural Gradient Descent for Shape Learningcited by 0
  10. 2026A Comparative Investigation of Thermodynamic Structure-Informed Neural Networkscited by 0
  11. 2026Direct From Darwin: Deriving Advanced Optimizers From Evolutionary First Principlescited by 0
  12. 2026Discrete energy as an exact label-free training objective for finite-element surrogatescited by 0
  13. 2026Finite basis physics-informed neural networks with hard constraints for viscous fluid flow in highly perforated domainscited by 0
  14. 2026DeepOHeat-v2: Self-Improving Operator Learning for Fast and Trustworthy Thermal Optimization in 3D-IC Designcited by 0
  15. 2026Derivative-Informed Training of Neural Operators On-the-Fly via Sketched Tangent Consistencycited by 0