# Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies

arXiv: https://arxiv.org/abs/2306.17648 · HTML: https://mlsanity.com/paper/2306.17648/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 83 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations](https://mlsanity.com/paper/2107.07871/) (2021)
- [Practical Gauss-Newton Optimisation for Deep Learning](https://mlsanity.com/paper/1706.03662/) (2017)
- [Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks](https://mlsanity.com/paper/2104.08426/) (2021)
- [Parallel Physics-Informed Neural Networks via Domain Decomposition](https://mlsanity.com/paper/2104.10013/) (2021)
- [Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating a class of inverse problems for PDEs](https://mlsanity.com/paper/2007.01138/) (2020)
- [When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?](https://mlsanity.com/paper/2109.09444/) (2021)
- [D3M: A deep domain decomposition method for partial differential equations](https://mlsanity.com/paper/1909.12236/) (2019)
- [Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology](https://mlsanity.com/paper/2211.08939/) (2022)
- [Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint](https://mlsanity.com/paper/1912.04862/) (2019)
- [PFNN: A Penalty-Free Neural Network Method for Solving a Class of Second-Order Boundary-Value Problems on Complex Geometries](https://mlsanity.com/paper/2004.06490/) (2020)
- [Layer-Parallel Training of Deep Residual Neural Networks](https://mlsanity.com/paper/1812.04352/) (2018)
- [Distributed physics informed neural network for data-efficient solution to partial differential equations](https://mlsanity.com/paper/1907.08967/) (2019)
- [Large-scale Neural Solvers for Partial Differential Equations](https://mlsanity.com/paper/2009.03730/) (2020)
- [Partition of unity networks: deep hp-approximation](https://mlsanity.com/paper/2101.11256/) (2021)
- [Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection](https://mlsanity.com/paper/2007.13171/) (2020)
- [Multilevel Minimization for Deep Residual Networks](https://mlsanity.com/paper/2004.06196/) (2020)
- [Globally Convergent Multilevel Training of Deep Residual Networks](https://mlsanity.com/paper/2107.07572/) (2021)
- [Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach](https://mlsanity.com/paper/2302.13143/) (2023)
- [A coarse space acceleration of deep-DDM](https://mlsanity.com/paper/2112.03732/) (2021)

## Built on by (most cited first, 20 of 20)

- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Residual-based attention in physics-informed neural networks](https://mlsanity.com/paper/2307.00379/) (2023)
- [Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?](https://mlsanity.com/paper/2405.04230/) (2024)
- [An operator preconditioning perspective on training in physics-informed machine learning](https://mlsanity.com/paper/2310.05801/) (2023)
- [Machine learning and domain decomposition methods -- a survey](https://mlsanity.com/paper/2312.14050/) (2023)
- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](https://mlsanity.com/paper/2403.18494/) (2024)
- [Learning a Neural Solver for Parametric PDE to Enhance Physics-Informed Methods](https://mlsanity.com/paper/2410.06820/) (2024)
- [Initialization-enhanced Physics-Informed Neural Network with Domain Decomposition (IDPINN)](https://mlsanity.com/paper/2406.03172/) (2024)
- [Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Method](https://mlsanity.com/paper/2508.00101/) (2025)
- [Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications](https://mlsanity.com/paper/2406.10997/) (2024)
- [A Nonoverlapping Domain Decomposition Method for Extreme Learning Machines: Elliptic Problems](https://mlsanity.com/paper/2406.15959/) (2024)
- [AB-PINNs: Adaptive-Basis Physics-Informed Neural Networks for Residual-Driven Domain Decomposition](https://mlsanity.com/paper/2510.08924/) (2025)
- [Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization](https://mlsanity.com/paper/2507.11513/) (2025)
- [Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods](https://mlsanity.com/paper/2312.13677/) (2023)
- [Preconditioned FEM-based Neural Networks for Solving Incompressible Fluid Flows and Related Inverse Problems](https://mlsanity.com/paper/2409.04067/) (2024)
- [A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations](https://mlsanity.com/paper/2411.06278/) (2024)
- [Multi-Preconditioned LBFGS for Training Finite-Basis PINNs](https://mlsanity.com/paper/2601.08709/) (2026)
- [Sparse Random-Feature Neural Networks with Krylov-Based SVD for Singularly Perturbed ODE](https://mlsanity.com/paper/2605.07286/) (2026)
- [Smooth Piecewise Cutting for Neural Operator to Handle Discontinuities and Sharp Transitions](https://mlsanity.com/paper/2605.19823/) (2026)
- [Two-level domain-decomposition AdaGrad method for scalable training of graph neural networks](https://mlsanity.com/paper/2608.22575/) (2026)

## Abstract (arXiv)

We propose to enhance the training of physics-informed neural networks (PINNs). To this aim, we introduce nonlinear additive and multiplicative preconditioning strategies for the widely used L-BFGS optimizer. The nonlinear preconditioners are constructed by utilizing the Schwarz domain-decomposition framework, where the parameters of the network are decomposed in a layer-wise manner. Through a series of numerical experiments, we demonstrate that both, additive and multiplicative preconditioners significantly improve the convergence of the standard L-BFGS optimizer, while providing more accurate solutions of the underlying partial differential equations. Moreover, the additive preconditioner is inherently parallel, thus giving rise to a novel approach to model parallelism.