# FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2505.10949 · HTML: https://mlsanity.com/paper/2505.10949/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.11833/) (2023)
- [Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs](https://mlsanity.com/paper/2101.08366/) (2021)
- [Parameterized Physics-informed Neural Networks for Parameterized PDEs](https://mlsanity.com/paper/2408.09446/) (2024)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [Is L2 Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?](https://mlsanity.com/paper/2206.02016/) (2022)
- [Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradients](https://mlsanity.com/paper/2209.07075/) (2022)
- [Critical Investigation of Failure Modes in Physics-informed Neural Networks](https://mlsanity.com/paper/2206.09961/) (2022)
- [PARCv2: Physics-aware Recurrent Convolutional Neural Networks for Spatiotemporal Dynamics Modeling](https://mlsanity.com/paper/2402.12503/) (2024)
- [Sub-Sequential Physics-Informed Learning with State Space Model](https://mlsanity.com/paper/2502.00318/) (2025)
- [ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks](https://mlsanity.com/paper/2502.00803/) (2025)

## Built on by (most cited first, 12 of 12)

- [Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.05230/) (2026)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [When PINNs Go Wrong: Pseudo-Time Stepping Against Spurious Solutions](https://mlsanity.com/paper/2604.23528/) (2026)
- [Physics-Informed Modeling for Wood Thermal Analysis and Prediction](https://mlsanity.com/paper/2606.23402/) (2026)
- [Estimating Parameter Fields in Multi-Physics PDEs from Scarce Measurements](https://mlsanity.com/paper/2509.00203/) (2025)
- [PINNs Failure Modes are Overfitting](https://mlsanity.com/paper/2605.30910/) (2026)
- [Physics-Informed Neural Network with Squeeze-Excitation-like Attention](https://mlsanity.com/paper/2606.19853/) (2026)
- [Tensor Train Diffusion: Leveraging Low-Rank Structures for High-Dimensional Score-Based Sampling](https://mlsanity.com/paper/2607.06841/) (2026)
- [Neither Precision Nor Architecture Alone: Controlled Tests of Failure Remedies for Physics-Informed Neural Networks](https://mlsanity.com/paper/2608.25327/) (2026)
- [SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning](https://mlsanity.com/paper/2608.29448/) (2026)
- [SoftServe: A Scalable Quasi-Newton Method for Deep Learning](https://mlsanity.com/paper/2610.02182/) (2026)
- [Transferability of Learned States in Neural PDE Solvers](https://mlsanity.com/paper/2610.10972/) (2026)

## Abstract (arXiv)

Physics Informed Neural Networks (PINNs) often exhibit failure modes in which the PDE residual loss converges while the solution error stays large, a phenomenon traditionally blamed on local optima separated from the true solution by steep loss barriers. We challenge this understanding by demonstrate that the real culprit is insufficient arithmetic precision: with standard FP32, the LBFGS optimizer prematurely satisfies its convergence test, freezing the network in a spurious failure phase. Simply upgrading to FP64 rescues optimization, enabling vanilla PINNs to solve PDEs without any failure modes. These results reframe PINN failure modes as precision induced stalls rather than inescapable local minima and expose a three stage training dynamic unconverged, failure, success whose boundaries shift with numerical precision. Our findings emphasize that rigorous arithmetic precision is the key to dependable PDE solving with neural networks.