mlsanityData to 7 Oct 2026

Challenges in Training PINNs: A Loss Landscape Perspective

arXiv 2402.01868 ↗First posted 2 Feb 2024cs.LG math.OC stat.MLcited by 109

Not on the map. Why?

Abstract (arXiv)

This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

41 of 72 references.

References · 41

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  3. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  6. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  7. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  8. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  9. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
  10. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  11. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  12. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
29 more
  1. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  2. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  3. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  4. 2021Physics-informed neural networks with hard constraints for inverse designcited by 277
  5. 2020hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decompositioncited by 236
  6. 2022A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networkscited by 233
  7. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  8. 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 220
  9. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2022Respecting causality is all you need for training physics-informed neural networkscited by 154
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  13. 2021Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equationscited by 142
  14. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  15. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  16. 2020Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEscited by 115
  17. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  18. 2021Efficient training of physics-informed neural networks via importance samplingcited by 109
  19. 2022Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transportcited by 92
  20. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  21. 2023PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEscited by 84
  22. 2017FALKON: An Optimal Large Scale Kernel Methodcited by 82
  23. 2021On the approximation of functions by tanh neural networkscited by 81
  24. 2015Input Sparsity Time Low-Rank Approximation via Ridge Leverage Score Samplingcited by 46
  25. 2019D3M: A deep domain decomposition method for partial differential equationscited by 44
  26. 2023An operator preconditioning perspective on training in physics-informed machine learningcited by 27
  27. 2017Oracle Complexity of Second-Order Methods for Smooth Convex Optimizationcited by 23
  28. 2022On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networkscited by 23
  29. 2023Aiming towards the minimizers: fast convergence of SGD for overparametrized problemscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 44; 2026: 44.

109 papers.

Sort:

By area:

  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  3. 2025Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networkscited by 41
  4. 2024On the expressiveness and spectral bias of KANscited by 36
  5. 2024Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?cited by 31
  6. 2024Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networkscited by 28
  7. 2024RoPINN: Region Optimized Physics-Informed Neural Networkscited by 26
  8. 2024A finite element-based physics-informed operator learning framework for spatiotemporal partial differential equations on arbitrary domainscited by 15
  9. 2025Physics-Informed Neural Networks with Hard Nonlinear Equality and Inequality Constraintscited by 14
  10. 2025Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomizationcited by 13
  11. 2024Geometry-Informed Neural Networkscited by 12
  12. 2024Physics-informed kernel learningcited by 11
38 more
  1. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  2. 2025FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networkscited by 11
  3. 2024Learning a Neural Solver for Parametric PDE to Enhance Physics-Informed Methodscited by 10
  4. 2025Enhancing Physics-Informed Neural Networks Through Feature Engineeringcited by 8
  5. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  6. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  7. 2025ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networkscited by 6
  8. 2025High precision PINNs in unbounded domains: application to singularity formulation in PDEscited by 6
  9. 2025PI-MFM: Physics-informed multimodal foundation model for solving partial differential equationscited by 6
  10. 2026Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operatorscited by 6
  11. 2024Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysiscited by 5
  12. 2024Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programmingcited by 4
  13. 2025SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learningcited by 4
  14. 2025Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraintscited by 4
  15. 2025Physics-Informed Machine Learning in Biomedical Science and Engineeringcited by 4
  16. 2026Out-of-Distribution Generalization for Neural Physics Solverscited by 4
  17. 2025MultiPDENet: PDE-embedded Learning with Multi-time-stepping for Accelerated Flow Simulationcited by 3
  18. 2025A Control Perspective on Training PINNscited by 3
  19. 2025Integration Matters for Learning PDEs with Backward SDEscited by 3
  20. 2025Sobolev acceleration for neural networkscited by 3
  21. 2025DAE-HardNet: A Physics Constrained Neural Network Enforcing Differential-Algebraic Hard Constraintscited by 3
  22. 2024Convergence of the Deep Galerkin Method for Finite State Mean Field Control Problemscited by 2
  23. 2024A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equationscited by 2
  24. 2024A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networkscited by 2
  25. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 2
  26. 2025Adaptive Collocation Point Strategies For Physics Informed Neural Networks via the QR Discrete Empirical Interpolation Methodcited by 2
  27. 2025Learn Singularly Perturbed Solutions via Homotopy Dynamicscited by 2
  28. 2025Physics-informed neural network estimation of active material properties in time-dependent cardiac biomechanical modelscited by 2
  29. 2025FMEnets: Flow, Material, and Energy networks for non-ideal plug flow reactor designcited by 2
  30. 2025Leveraging Influence Functions for Resampling Data in Physics-Informed Neural Networkscited by 2
  31. 2025MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solvingcited by 2
  32. 2025Bayesian BiLO: Bilevel Local Operator Learning for Efficient Uncertainty Quantification of Bayesian PDE Inverse Problems with Low-Rank Adaptationcited by 2
  33. 2025Revisiting Heat Flux Analysis of Tungsten Monoblock Divertor on EAST using Physics-Informed Neural Networkcited by 2
  34. 2026Parametrization of subgrid scales in long-term simulations of the shallow-water equations using machine learning and convex limitingcited by 2
  35. 2026Rethinking Input Domains in Physics-Informed Neural Networks via Geometric Compactification Mappingscited by 2
  36. 2024Towards Foundation Model for Chemical Reactor Modeling: Meta-Learning with Physics-Informed Adaptationcited by 1
  37. 2024PINNfluence: Interpreting PINNs through Influence Functionscited by 1
  38. 2025Damper-B-PINN: Damper Characteristics-Based Bayesian Physics-Informed Neural Network for Vehicle State Estimationcited by 1

50 most cited of 109 All 109 (JSON)