mlsanityData to 7 Oct 2026

Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks

arXiv 2501.16371 ↗First posted 22 Jan 2025cs.LG cs.AI math.OCcited by 41

Not on the map. Why?

Abstract (arXiv)

Physics-Informed Neural Networks (PINNs) have revolutionized the computation of PDE solutions by integrating partial differential equations (PDEs) into the neural network's training process as soft constraints, becoming an important component of the scientific machine learning (SciML) ecosystem. More recently, physics-informed Kolmogorv-Arnold networks (PIKANs) have also shown to be effective and comparable in accuracy with PINNs. In their current implementation, both PINNs and PIKANs are mainly optimized using first-order methods like Adam, as well as quasi-Newton methods such as BFGS and its low-memory variant, L-BFGS. However, these optimizers often struggle with highly non-linear and non-convex loss landscapes, leading to challenges such as slow convergence, local minima entrapment, and (non)degenerate saddle points. In this study, we investigate the performance of Self-Scaled BFGS (SSBFGS), Self-Scaled Broyden (SSBroyden) methods and other advanced quasi-Newton schemes, including BFGS and L-BFGS with different line search strategies. These methods dynamically rescale updates based on historical gradient information, thus enhancing training efficiency and accuracy. We systematically compare these optimizers using both PINNs and PIKANs on key challenging PDEs, including the Burgers, Allen-Cahn, Kuramoto-Sivashinsky, Ginzburg-Landau, and Stokes equations. Additionally, we evaluate the performance of SSBFGS and SSBroyden for Deep Operator Network (DeepONet) architectures, demonstrating their effectiveness for data-driven operator learning. Our findings provide state-of-the-art results with orders-of-magnitude accuracy improvements without the use of adaptive weights or any other enhancements typically employed in PINNs.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 44 of its 97 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

44 of 97 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  7. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2020B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Datacited by 329
  11. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  12. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
32 more
  1. 2020hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decompositioncited by 236
  2. 2016Deep Learning without Poor Local Minimacited by 233
  3. 2022A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networkscited by 233
  4. 2020Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanismcited by 231
  5. 2020PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domaincited by 173
  6. 2022Respecting causality is all you need for training physics-informed neural networkscited by 154
  7. 2018Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problemscited by 144
  8. 2017Machine Learning of Linear Differential Equations using Gaussian Processescited by 139
  9. 2021Exact imposition of boundary conditions with distance functions in physics-informed deep neural networkscited by 135
  10. 2021Parallel Physics-Informed Neural Networks via Domain Decompositioncited by 112
  11. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
  12. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 103
  13. 2020Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating a class of inverse problems for PDEscited by 101
  14. 2021PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEscited by 97
  15. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  16. 2020A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networkscited by 86
  17. 2024PirateNets: Physics-informed Deep Learning with Residual Adaptive Networkscited by 70
  18. 2019Physics Informed Extreme Learning Machine (PIELM) -- A rapid method for the numerical solution of partial differential equationscited by 66
  19. 2021Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEscited by 56
  20. 2020On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEscited by 31
  21. 2024Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?cited by 31
  22. 2016A Multi-Batch L-BFGS Method for Machine Learningcited by 30
  23. 2024Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamicscited by 27
  24. 2023A Framework Based on Symbolic Regression Coupled with eXtended Physics-Informed Neural Networks for Gray-Box Learning of Equations of Motion from Datacited by 18
  25. 2024Optimistix: modular optimisation in JAX and Equinoxcited by 17
  26. 2024Learning in PINNs: Phase transition, diffusion equilibrium, and generalizationcited by 13
  27. 2024Predicting Crack Nucleation and Propagation in Brittle Materials Using Deep Operator Networks with Diverse Trunk Architecturescited by 12
  28. 2022Revisiting PINNs: Generative Adversarial Physics-informed Neural Networks and Point-weighting Methodcited by 10
  29. 2018Structured Quasi-Newton Methods for Optimization with Orthogonality Constraintscited by 7
  30. 2019Derivative-free optimization methodscited by 6
  31. 2022Error-Aware B-PINNs: Improving Uncertainty Quantification in Bayesian Physics-Informed Neural Networkscited by 4
  32. 2023Characterization of partial wetting by CMAS droplets using multiphase many-body dissipative particle dynamics and data-driven discovery based on PINNscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 26.

41 papers.

Sort:

By area:

  1. 2025A Practitioner's Guide to Kolmogorov-Arnold Networkscited by 16
  2. 2026Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networkscited by 11
  3. 2025Representation Meets Optimization: Training PINNs and PIKANs for Gray-Box Discovery in Systems Pharmacologycited by 10
  4. 2025ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithmscited by 7
  5. 2025High precision PINNs in unbounded domains: application to singularity formulation in PDEscited by 6
  6. 2025A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learningcited by 6
  7. 2026Spectral bias in physics-informed and operator learning: Analysis and mitigation guidelinescited by 6
  8. 2026Inelastic Constitutive Kolmogorov-Arnold Networks: A generalized framework for automated discovery of interpretable inelastic material modelscited by 5
  9. 2025Enforcing hidden physics in physics-informed neural networkscited by 3
  10. 2026PINNs in PDE Constrained Optimal Control Problems: Direct vs Indirect Methodscited by 3
  11. 2025Training Deep Physics-Informed Kolmogorov-Arnold Networkscited by 2
  12. 2026Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithmcited by 2
29 more
  1. 2026Self-Scaled Broyden Family of Quasi-Newton Methods in JAXcited by 2
  2. 2026Tackling Failure Modes of PINNs and PIKANs Using Conflict-Free Gradientscited by 2
  3. 2025Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebracited by 1
  4. 2025Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectivescited by 1
  5. 2025Boundary condition enforcement with PINNs: a comparative study and verification on 3D geometriescited by 1
  6. 2026Visualizing the loss landscapes of physics-informed neural networkscited by 1
  7. 2026Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimizationcited by 1
  8. 2026Lightweight Geometric Adaptation for Training Physics-Informed Neural Networkscited by 1
  9. 2026An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networkscited by 1
  10. 2025Mask-PINNs: Mitigating Internal Covariate Shift in Physics-Informed Neural Networkscited by 0
  11. 2025Crack Path Prediction with Operator Learning using Discrete Particle System data Generationcited by 0
  12. 2025Forecasting India's Demographic Transition Under Fertility Policy Scenarios Using hybrid LSTM-PINN Modelcited by 0
  13. 2025ASPEN: An Adaptive Spectral Physics-Enabled Network for Ginzburg-Landau Dynamicscited by 0
  14. 2025GIMLET: Generalizable and Interpretable Model Learning through Embedded Thermodynamicscited by 0
  15. 2026Drug Release Modeling using Physics-Informed Neural Networkscited by 0
  16. 2026Pseudo-differential-enhanced physics-informed neural networkscited by 0
  17. 2026Stability and Bifurcation Analysis of Nonlinear PDEs via Random Projection-based PINNs: A Krylov-Arnoldi Approachcited by 0
  18. 2026YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perceptioncited by 0
  19. 2026Residual Attention Physics-Informed Neural Networks for Robust Multiphysics Simulation of Steady-State Electrothermal Energy Systemscited by 0
  20. 2026Direct From Darwin: Deriving Advanced Optimizers From Evolutionary First Principlescited by 0
  21. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
  22. 2026Oscillatory State-Space Models as Inductive Biases for Physics-Informed Neural PDE Solverscited by 0
  23. 2026Curvature-aware dynamic precision approach for physics-informed neural networkscited by 0
  24. 2026Accelerating SAV-based optimization via randomized low-rank Hessian approximationcited by 0
  25. 2026Adjoint Method versus Physics-Informed Neural Networks in PDE-Constrained Inverse Problemscited by 0
  26. 2026From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNscited by 0
  27. 2026SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learningcited by 0
  28. 2026Cost-Accuracy Trade-offs: Neural Operator vs Classical Numerical Solvercited by 0
  29. 2026AURA: Angular Update Rate Adaptation for training complex-valued neural networkscited by 0