# An operator preconditioning perspective on training in physics-informed machine learning

arXiv: https://arxiv.org/abs/2310.05801 · HTML: https://mlsanity.com/paper/2310.05801/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Convolutional Neural Operators for robust and accurate learning of PDEs](https://mlsanity.com/paper/2302.01178/) (2023)
- [Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1708.00588/) (2017)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Variational Physics-Informed Neural Networks For Solving Partial Differential Equations](https://mlsanity.com/paper/1912.00873/) (2019)
- [Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations](https://mlsanity.com/paper/2107.07871/) (2021)
- [On the linearity of large non-linear models: when and why the tangent kernel is constant](https://mlsanity.com/paper/2010.01092/) (2020)
- [Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEs](https://mlsanity.com/paper/2006.16144/) (2020)
- [Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs](https://mlsanity.com/paper/2106.14473/) (2021)
- [Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations](https://mlsanity.com/paper/2203.01360/) (2022)
- [Physics Informed Neural Networks for Simulating Radiative Transfer](https://mlsanity.com/paper/2009.13291/) (2020)
- [wPINNs: Weak Physics informed neural networks for approximating entropy solutions of hyperbolic conservation laws](https://mlsanity.com/paper/2207.08483/) (2022)
- [Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies](https://mlsanity.com/paper/2306.17648/) (2023)
- [Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equations](https://mlsanity.com/paper/2305.06000/) (2023)

## Built on by (most cited first, 27 of 27)

- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [Numerical analysis of physics-informed neural networks and related models in physics-informed machine learning](https://mlsanity.com/paper/2402.10926/) (2024)
- [Parameterized Physics-informed Neural Networks for Parameterized PDEs](https://mlsanity.com/paper/2408.09446/) (2024)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Dynamical Measure Transport and Neural PDE Solvers for Sampling](https://mlsanity.com/paper/2407.07873/) (2024)
- [Geometry-Informed Neural Networks](https://mlsanity.com/paper/2402.14009/) (2024)
- [PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations](https://mlsanity.com/paper/2412.05994/) (2024)
- [Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEs](https://mlsanity.com/paper/2505.03595/) (2025)
- [Transfer learning for improved generalizability in causal physics-informed neural networks for beam simulations](https://mlsanity.com/paper/2311.00578/) (2023)
- [Enhancing Physics-Informed Neural Networks Through Feature Engineering](https://mlsanity.com/paper/2502.07209/) (2025)
- [ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning](https://mlsanity.com/paper/2412.10782/) (2024)
- [Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.21404/) (2025)
- [Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications](https://mlsanity.com/paper/2406.10997/) (2024)
- [Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics](https://mlsanity.com/paper/2507.02748/) (2025)
- [A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks](https://mlsanity.com/paper/2412.09752/) (2024)
- [Beyond Accuracy: EcoL2 Metric for Sustainable Neural PDE Solvers](https://mlsanity.com/paper/2505.12556/) (2025)
- [Leveraging Influence Functions for Resampling Data in Physics-Informed Neural Networks](https://mlsanity.com/paper/2506.16443/) (2025)
- [PINNfluence: Interpreting PINNs through Influence Functions](https://mlsanity.com/paper/2409.08958/) (2024)
- [Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations](https://mlsanity.com/paper/2503.21166/) (2025)
- [Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2505.11638/) (2025)
- [Gauss-Newton Natural Gradient Descent for Shape Learning](https://mlsanity.com/paper/2602.00099/) (2026)
- [A Comparative Investigation of Thermodynamic Structure-Informed Neural Networks](https://mlsanity.com/paper/2603.26803/) (2026)
- [Direct From Darwin: Deriving Advanced Optimizers From Evolutionary First Principles](https://mlsanity.com/paper/2605.05284/) (2026)
- [Discrete energy as an exact label-free training objective for finite-element surrogates](https://mlsanity.com/paper/2608.05437/) (2026)
- [Finite basis physics-informed neural networks with hard constraints for viscous fluid flow in highly perforated domains](https://mlsanity.com/paper/2608.08114/) (2026)
- [DeepOHeat-v2: Self-Improving Operator Learning for Fast and Trustworthy Thermal Optimization in 3D-IC Design](https://mlsanity.com/paper/2608.16080/) (2026)
- [Derivative-Informed Training of Neural Operators On-the-Fly via Sketched Tangent Consistency](https://mlsanity.com/paper/2609.32797/) (2026)

## Abstract (arXiv)

In this paper, we investigate the behavior of gradient descent algorithms in physics-informed machine learning methods like PINNs, which minimize residuals connected to partial differential equations (PDEs). Our key result is that the difficulty in training these models is closely related to the conditioning of a specific differential operator. This operator, in turn, is associated to the Hermitian square of the differential operator of the underlying PDE. If this operator is ill-conditioned, it results in slow or infeasible training. Therefore, preconditioning this operator is crucial. We employ both rigorous mathematical analysis and empirical evaluations to investigate various strategies, explaining how they better condition this critical operator, and consequently improve training.