# Characterizing possible failure modes in physics-informed neural networks

arXiv: https://arxiv.org/abs/2109.01050 · HTML: https://mlsanity.com/paper/2109.01050/

## Key facts

- Cited by 643 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data](https://mlsanity.com/paper/1901.06314/) (2019)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [DC3: A learning method for optimization with hard constraints](https://mlsanity.com/paper/2104.12225/) (2021)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks](https://mlsanity.com/paper/1906.05747/) (2019)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework](https://mlsanity.com/paper/2012.07938/) (2020)
- [Imposing Hard Constraints on Deep Networks: Promises and Limitations](https://mlsanity.com/paper/1706.02025/) (2017)
- [Distributed physics informed neural network for data-efficient solution to partial differential equations](https://mlsanity.com/paper/1907.08967/) (2019)

## Built on by (most cited first, 30 of 643)

- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Neural Operators for Accelerating Scientific Simulations and Design](https://mlsanity.com/paper/2309.15325/) (2023)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks](https://mlsanity.com/paper/2406.02917/) (2024)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations](https://mlsanity.com/paper/2407.07218/) (2024)
- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [When Do Flat Minima Optimizers Work?](https://mlsanity.com/paper/2202.00661/) (2022)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Residual-based attention in physics-informed neural networks](https://mlsanity.com/paper/2307.00379/) (2023)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [Can Physics-Informed Neural Networks beat the Finite Element Method?](https://mlsanity.com/paper/2302.04107/) (2023)
- [Separable Physics-Informed Neural Networks](https://mlsanity.com/paper/2306.15969/) (2023)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.11833/) (2023)
- [Learning Physical Models that Can Respect Conservation Laws](https://mlsanity.com/paper/2302.11002/) (2023)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints](https://mlsanity.com/paper/2506.04171/) (2025)
- [A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions](https://mlsanity.com/paper/2302.14227/) (2023)
- [Multi-stage Neural Networks: Function Approximator of Machine Precision](https://mlsanity.com/paper/2307.08934/) (2023)
- [Investigating Compounding Prediction Errors in Learned Dynamics Models](https://mlsanity.com/paper/2203.09637/) (2022)

## Abstract (arXiv)

Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.