# Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?

arXiv: https://arxiv.org/abs/2405.04230 · HTML: https://mlsanity.com/paper/2405.04230/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 93 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks](https://mlsanity.com/paper/2104.08426/) (2021)
- [Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks](https://mlsanity.com/paper/2007.04542/) (2020)
- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networks](https://mlsanity.com/paper/2007.07442/) (2020)
- [Tackling the Curse of Dimensionality with Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.12306/) (2023)
- [Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks](https://mlsanity.com/paper/1905.04817/) (2019)
- [Physics-informed neural networks for inverse problems in supersonic flows](https://mlsanity.com/paper/2202.11821/) (2022)
- [Residual-based attention in physics-informed neural networks](https://mlsanity.com/paper/2307.00379/) (2023)
- [Error estimates for physics informed neural networks approximating the Navier-Stokes equations](https://mlsanity.com/paper/2203.09346/) (2022)
- [Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs](https://mlsanity.com/paper/2106.14473/) (2021)
- [A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture](https://mlsanity.com/paper/2005.00615/) (2020)
- [Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology](https://mlsanity.com/paper/2211.08939/) (2022)
- [Physics Informed Neural Networks for Simulating Radiative Transfer](https://mlsanity.com/paper/2009.13291/) (2020)
- [Accelerated Training of Physics-Informed Neural Networks (PINNs) using Meshless Discretizations](https://mlsanity.com/paper/2205.09332/) (2022)
- [Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies](https://mlsanity.com/paper/2306.17648/) (2023)
- [Improving physics-informed neural networks with meta-learned optimization](https://mlsanity.com/paper/2303.07127/) (2023)
- [A practical PINN framework for multi-scale problems with multi-magnitude loss terms](https://mlsanity.com/paper/2308.06672/) (2023)

## Built on by (most cited first, 30 of 31)

- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.16371/) (2025)
- [Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.05230/) (2026)
- [Representation Meets Optimization: Training PINNs and PIKANs for Gray-Box Discovery in Systems Pharmacology](https://mlsanity.com/paper/2504.07379/) (2025)
- [Spectral bias in physics-informed and operator learning: Analysis and mitigation guidelines](https://mlsanity.com/paper/2602.19265/) (2026)
- [High precision PINNs in unbounded domains: application to singularity formulation in PDEs](https://mlsanity.com/paper/2506.19243/) (2025)
- [ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2512.03476/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [Physics-Informed Machine Learning in Biomedical Science and Engineering](https://mlsanity.com/paper/2510.05433/) (2025)
- [Enforcing hidden physics in physics-informed neural networks](https://mlsanity.com/paper/2511.14348/) (2025)
- [Hard-Constrained Neural Networks with Physics-Embedded Architecture for Residual Dynamics Learning and Invariant Enforcement in Cyber-Physical Systems](https://mlsanity.com/paper/2511.23307/) (2025)
- [Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation](https://mlsanity.com/paper/2508.21571/) (2025)
- [Physics-Informed Neural Networks with Fourier Features and Attention-Driven Decoding](https://mlsanity.com/paper/2510.05385/) (2025)
- [Convergence and Sketching-Based Efficient Computation of Neural Tangent Kernel Weights in Physics-Based Loss](https://mlsanity.com/paper/2511.15530/) (2025)
- [Boundary condition enforcement with PINNs: a comparative study and verification on 3D geometries](https://mlsanity.com/paper/2512.14941/) (2025)
- [Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2505.11638/) (2025)
- [Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectives](https://mlsanity.com/paper/2510.11987/) (2025)
- [Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.15392/) (2026)
- [An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks](https://mlsanity.com/paper/2607.02194/) (2026)
- [Estimating Parameter Fields in Multi-Physics PDEs from Scarce Measurements](https://mlsanity.com/paper/2509.00203/) (2025)
- [Seeing Structural Failure Before it Happens: An Image-Based Physics-Informed Neural Network (PINN) for Spaghetti Bridge Load Prediction](https://mlsanity.com/paper/2510.23117/) (2025)
- [Residual Attention Physics-Informed Neural Networks for Robust Multiphysics Simulation of Steady-State Electrothermal Energy Systems](https://mlsanity.com/paper/2603.23578/) (2026)
- [Meta-Inverse Physics-Informed Neural Networks for High-Dimensional Ordinary Differential Equations](https://mlsanity.com/paper/2605.03511/) (2026)
- [Curvature-aware dynamic precision approach for physics-informed neural networks](https://mlsanity.com/paper/2606.04736/) (2026)
- [Adjoint Method versus Physics-Informed Neural Networks in PDE-Constrained Inverse Problems](https://mlsanity.com/paper/2606.12337/) (2026)
- [From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs](https://mlsanity.com/paper/2608.04206/) (2026)
- [Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network](https://mlsanity.com/paper/2608.10351/) (2026)
- [SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning](https://mlsanity.com/paper/2608.29448/) (2026)
- [Direct Optimization of a 3D Finite-Source Reflector via Neural-Network Parameterization](https://mlsanity.com/paper/2609.00899/) (2026)

## Abstract (arXiv)

This study investigates the potential accuracy boundaries of physics-informed neural networks, contrasting their approach with previous similar works and traditional numerical methods. We find that selecting improved optimization algorithms significantly enhances the accuracy of the results. Simple modifications to the loss function may also improve precision, offering an additional avenue for enhancement. Despite optimization algorithms having a greater impact on convergence than adjustments to the loss function, practical considerations often favor tweaking the latter due to ease of implementation. On a global scale, the integration of an enhanced optimizer and a marginally adjusted loss function enables a reduction in the loss function by several orders of magnitude across diverse physical problems. Consequently, our results obtained using compact networks (typically comprising 2 or 3 layers of 20-30 neurons) achieve accuracies comparable to finite difference schemes employing thousands of grid points. This study encourages the continued advancement of PINNs and associated optimization techniques for broader applications across various fields.