mlsanityData to 7 Oct 2026

Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications

arXiv 2406.10997 ↗First posted 16 Jun 2024math.NA cs.LG cs.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce a novel two-level overlapping additive Schwarz preconditioner for accelerating the training of scientific machine learning applications. The design of the proposed preconditioner is motivated by the nonlinear two-level overlapping additive Schwarz preconditioner. The neural network parameters are decomposed into groups (subdomains) with overlapping regions. In addition, the network's feed-forward structure is indirectly imposed through a novel subdomain-wise synchronization strategy and a coarse-level training step. Through a series of numerical experiments, which consider physics-informed neural networks and operator learning approaches, we demonstrate that the proposed two-level preconditioner significantly speeds up the convergence of the standard (LBFGS) optimizer while also yielding more accurate machine learning models. Moreover, the devised preconditioner is designed to take advantage of model-parallel computations, which can further reduce the training time.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 27 of its 71 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

27 of 71 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  7. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  8. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  9. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  10. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
  11. 2021Physics-informed neural networks with hard constraints for inverse designcited by 277
  12. 2014One weird trick for parallelizing convolutional neural networkscited by 264
15 more
  1. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  2. 2021Exact imposition of boundary conditions with distance functions in physics-informed deep neural networkscited by 135
  3. 2020Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating a class of inverse problems for PDEscited by 101
  4. 2023Residual-based attention in physics-informed neural networkscited by 72
  5. 2022Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodologycited by 43
  6. 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
  7. 2023An operator preconditioning perspective on training in physics-informed machine learningcited by 27
  8. 2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
  9. 2020Multilevel Minimization for Deep Residual Networkscited by 11
  10. 2023Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtrackingcited by 9
  11. 2023Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Trainingcited by 6
  12. 2024Activations and Gradients Compression for Model-Parallel Trainingcited by 6
  13. 2021Parareal Neural Networks Emulating a Parallel-in-time Algorithmcited by 4
  14. 2021Two-level Group Convolutioncited by 4
  15. 2022Nonlinear Schwarz preconditioning for nonlinear optimization problems with bound constraintscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Methodcited by 6
  2. 2025Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimizationcited by 3
  3. 2025Hybrid Iterative Solvers with Geometry-Aware Neural Preconditioners for Parametric PDEscited by 3
  4. 2026Multi-Preconditioned LBFGS for Training Finite-Basis PINNscited by 1
  5. 2026Two-level domain-decomposition AdaGrad method for scalable training of graph neural networkscited by 0