# Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2306.09478 · HTML: https://mlsanity.com/paper/2306.09478/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems](https://mlsanity.com/paper/1809.08327/) (2018)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1709.01953/) (2017)
- [Long-time integration of parametric evolution equations with physics-informed DeepONets](https://mlsanity.com/paper/2106.05384/) (2021)
- [Reliable extrapolation of deep neural operators informed by physics or sparse observations](https://mlsanity.com/paper/2212.06347/) (2022)
- [DPM: A Novel Training Method for Physics-Informed Neural Networks in Extrapolation](https://mlsanity.com/paper/2012.02681/) (2020)
- [Neural Operator Learning for Long-Time Integration in Dynamical Systems with Recurrent Neural Networks](https://mlsanity.com/paper/2303.02243/) (2023)
- [The Wasserstein-Fourier Distance for Stationary Time Series](https://mlsanity.com/paper/1912.05509/) (2019)
- [Transfer Learning with Physics-Informed Neural Networks for Efficient Simulation of Branched Flows](https://mlsanity.com/paper/2211.00214/) (2022)
- [Physics-Informed Echo State Networks for Chaotic Systems Forecasting](https://mlsanity.com/paper/1906.11122/) (2019)
- [SMS: Spiking Marching Scheme for Efficient Long Time Integration of Differential Equations](https://mlsanity.com/paper/2211.09928/) (2022)

## Built on by (most cited first, 9 of 9)

- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Neural oscillators for generalization of physics-informed machine learning](https://mlsanity.com/paper/2308.08989/) (2023)
- [MoGERNN: An Inductive Traffic Predictor for Unobserved Locations](https://mlsanity.com/paper/2501.12281/) (2025)
- [Improving physics-informed neural network extrapolation via transfer learning and adaptive activation functions](https://mlsanity.com/paper/2507.12659/) (2025)
- [xLSTM-PINN: Memory-Gated Spectral Remodeling for Physics-Informed Learning](https://mlsanity.com/paper/2511.12512/) (2025)
- [Accelerating 4D Hyperspectral Imaging through Physics-Informed Neural Representation and Adaptive Sampling](https://mlsanity.com/paper/2604.06561/) (2026)
- [SPLIT-PINN: Separable Probability Learning Technique via Physics-Informed Neural Networks for High-Dimensional Probabilistic Modeling](https://mlsanity.com/paper/2606.04000/) (2026)
- [Wrong Operator or Blind Design? A Post-Training Diagnostic for Physics-Informed Coefficient Learning](https://mlsanity.com/paper/2608.16925/) (2026)
- [Into the danger zone: stable extrapolation in high-dimensional function and operator learning](https://mlsanity.com/paper/2609.36709/) (2026)

## Abstract (arXiv)

Physics-informed Neural Networks (PINNs) have recently gained popularity due to their effective approximation of partial differential equations (PDEs) using deep neural networks (DNNs). However, their out of domain behavior is not well understood, with previous work speculating that the presence of high frequency components in the solution function might be to blame for poor extrapolation performance. In this paper, we study the extrapolation behavior of PINNs on a representative set of PDEs of different types, including high-dimensional PDEs. We find that failure to extrapolate is not caused by high frequencies in the solution function, but rather by shifts in the support of the Fourier spectrum over time. We term these spectral shifts and quantify them by introducing a Weighted Wasserstein-Fourier distance (WWF). We show that the WWF can be used to predict PINN extrapolation performance, and that in the absence of significant spectral shifts, PINN predictions stay close to the true solution even in extrapolation. Finally, we propose a transfer learning-based strategy to mitigate the effects of larger spectral shifts, which decreases extrapolation errors by up to 82%.