# Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2604.05230 · HTML: https://mlsanity.com/paper/2604.05230/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 64 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Physics-informed neural networks for inverse problems in supersonic flows](https://mlsanity.com/paper/2202.11821/) (2022)
- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Separable Physics-Informed Neural Networks](https://mlsanity.com/paper/2306.15969/) (2023)
- [A Progressive Batching L-BFGS Method for Machine Learning](https://mlsanity.com/paper/1802.05374/) (2018)
- [Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.16371/) (2025)
- [Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?](https://mlsanity.com/paper/2405.04230/) (2024)
- [Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics](https://mlsanity.com/paper/2402.10680/) (2024)
- [A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctions](https://mlsanity.com/paper/2401.10190/) (2024)
- [Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective](https://mlsanity.com/paper/2402.03496/) (2024)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Position: Optimization in SciML Should Employ the Function Space Geometry](https://mlsanity.com/paper/2402.07318/) (2024)
- [A Framework Based on Symbolic Regression Coupled with eXtended Physics-Informed Neural Networks for Gray-Box Learning of Equations of Motion from Data](https://mlsanity.com/paper/2305.10706/) (2023)
- [Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization](https://mlsanity.com/paper/2505.12149/) (2025)
- [FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.10949/) (2025)
- [Probabilistic Deep Learning for Real-Time Large Deformation Simulations](https://mlsanity.com/paper/2111.01867/) (2021)
- [Invariance Properties of the Natural Gradient in Overparametrised Systems](https://mlsanity.com/paper/2206.15273/) (2022)
- [NekRS, a GPU-Accelerated Spectral Element Navier-Stokes Solver](https://mlsanity.com/paper/2104.05829/) (2021)
- [CMINNs: Compartment Model Informed Neural Networks -- Unlocking Drug Dynamics](https://mlsanity.com/paper/2409.12998/) (2024)
- [Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.21404/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)

## Built on by (most cited first, 11 of 11)

- [GRAFT-ATHENA: Self-Improving Agentic Teams for Autonomous Discovery and Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2605.11117/) (2026)
- [Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.15392/) (2026)
- [Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit](https://mlsanity.com/paper/2605.08352/) (2026)
- [Curvature-aware dynamic precision approach for physics-informed neural networks](https://mlsanity.com/paper/2606.04736/) (2026)
- [Wall Shear Stress Reconstruction from Concentration: Differentiable Physics and Physics-Informed Neural Networks](https://mlsanity.com/paper/2606.06313/) (2026)
- [Adjoint Method versus Physics-Informed Neural Networks in PDE-Constrained Inverse Problems](https://mlsanity.com/paper/2606.12337/) (2026)
- [From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs](https://mlsanity.com/paper/2608.04206/) (2026)
- [SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning](https://mlsanity.com/paper/2608.29448/) (2026)
- [A Systematic Analysis of Automatic Differentiation versus Discretization-based Constraints for Physics-Informed PDE Solvers](https://mlsanity.com/paper/2609.07437/) (2026)
- [Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers](https://mlsanity.com/paper/2609.20641/) (2026)
- [Cost-Accuracy Trade-offs: Neural Operator vs Classical Numerical Solver](https://mlsanity.com/paper/2609.24021/) (2026)

## Abstract (arXiv)

Efficient and robust optimization is essential for neural networks, enabling scientific machine learning models to converge rapidly to very high accuracy -- faithfully capturing complex physical behavior governed by differential equations. In this work, we present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs) for challenging partial (PDEs) and ordinary differential equations (ODEs). Specifically, we provide efficient implementations of the Natural Gradient (NG) optimizer, Self-Scaling BFGS and Broyden optimizers, and demonstrate their performance on problems including the Helmholtz equation, Stokes flow, inviscid Burgers equation, Euler equations for high-speed flows, and stiff ODEs arising in pharmacokinetics and pharmacodynamics. Beyond optimizer development, we also propose new PINN-based methods for solving the inviscid Burgers and Euler equations, and compare the resulting solutions against high-order numerical methods to provide a rigorous and fair assessment. Finally, we address the challenge of scaling these quasi-Newton optimizers for batched training, enabling efficient and scalable solutions for large data-driven problems.