# Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling

arXiv: https://arxiv.org/abs/2207.02338 · HTML: https://mlsanity.com/paper/2207.02338/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 14 of 14)

- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Efficient training of physics-informed neural networks via importance sampling](https://mlsanity.com/paper/2104.12325/) (2021)
- [Learning in Sinusoidal Spaces with Physics-Informed Neural Networks](https://mlsanity.com/paper/2109.09338/) (2021)
- [Physics-informed deep learning for incompressible laminar flows](https://mlsanity.com/paper/2002.10558/) (2020)
- [Machine learning for metal additive manufacturing: Predicting temperature and melt pool fluid dynamics using physics-informed neural networks](https://mlsanity.com/paper/2008.13547/) (2020)
- [On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2203.13648/) (2022)
- [Is L2 Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?](https://mlsanity.com/paper/2206.02016/) (2022)
- [Physics-informed neural networks for myocardial perfusion MRI quantification](https://mlsanity.com/paper/2011.12844/) (2020)
- [PCE-PINNs: Physics-Informed Neural Networks for Uncertainty Propagation in Ocean Modeling](https://mlsanity.com/paper/2105.02939/) (2021)
- [Revisit Geophysical Imaging in A New View of Physics-informed Generative Adversarial Learning](https://mlsanity.com/paper/2109.11452/) (2021)

## Built on by (most cited first, 30 of 73)

- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2408.11104/) (2024)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [Stacked networks improve physics-informed training: applications to neural networks and deep operator networks](https://mlsanity.com/paper/2311.06483/) (2023)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Separable DeepONet: Breaking the Curse of Dimensionality in Physics-Informed Machine Learning](https://mlsanity.com/paper/2407.15887/) (2024)
- [Knowledge-guided Machine Learning: Current Trends and Future Prospects](https://mlsanity.com/paper/2403.15989/) (2024)
- [FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.10949/) (2025)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Better Neural PDE Solvers Through Data-Free Mesh Movers](https://mlsanity.com/paper/2312.05583/) (2023)
- [Complex Physics-Informed Neural Network](https://mlsanity.com/paper/2502.04917/) (2025)
- [A physics-informed transformer neural operator for learning generalized solutions of initial boundary value problems](https://mlsanity.com/paper/2412.09009/) (2024)
- [ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks](https://mlsanity.com/paper/2502.00803/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [RBF-PINN: Non-Fourier Positional Embedding in Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.08367/) (2024)
- [Deep adaptive sampling for surrogate modeling without labeled data](https://mlsanity.com/paper/2402.11283/) (2024)
- [Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees](https://mlsanity.com/paper/2410.18153/) (2024)
- [Dual-Balancing for Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.11117/) (2025)
- [PTPI-DL-ROMs: pre-trained physics-informed deep learning-based reduced order models for nonlinear parametrized PDEs](https://mlsanity.com/paper/2405.08558/) (2024)
- [Beyond Derivative Pathology of PINNs: Variable Splitting Strategy with Convergence Analysis](https://mlsanity.com/paper/2409.20383/) (2024)
- [PINNverse: Accurate parameter estimation in differential equations from noisy data with constrained physics-informed neural networks](https://mlsanity.com/paper/2504.05248/) (2025)
- [RAMS: Residual-based adversarial-gradient moving sample method for scientific machine learning in solving partial differential equations](https://mlsanity.com/paper/2509.01234/) (2025)
- [AB-PINNs: Adaptive-Basis Physics-Informed Neural Networks for Residual-Driven Domain Decomposition](https://mlsanity.com/paper/2510.08924/) (2025)
- [Element-wise Multiplication Based Deeper Physics-Informed Neural Networks](https://mlsanity.com/paper/2406.04170/) (2024)
- [Solving Differential Equations with Constrained Learning](https://mlsanity.com/paper/2410.22796/) (2024)
- [Enhanced Vascular Flow Simulations in Aortic Aneurysm via Physics-Informed Neural Networks and Deep Operator Networks](https://mlsanity.com/paper/2503.17402/) (2025)
- [Integration Matters for Learning PDEs with Backward SDEs](https://mlsanity.com/paper/2505.01078/) (2025)
- [PRISMA: Improving the Accuracy-Latency Frontier of Diffusion-based PDE Solvers Using Physics-Informed Spectral Attention](https://mlsanity.com/paper/2512.01370/) (2025)
- [Investigating the Ability of PINNs To Solve Burgers' PDE Near Finite-Time BlowUp](https://mlsanity.com/paper/2310.05169/) (2023)
- [Feature Mapping in Physics-Informed Neural Networks (PINNs)](https://mlsanity.com/paper/2402.06955/) (2024)

## Abstract (arXiv)

Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing the "failure modes" of PINNs, although a thorough understanding of the connection between PINN failure modes and sampling strategies is missing. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that training PINNs relies on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures, characterized by highly imbalanced PDE residual fields. To mitigate propagation failures, we propose a novel Retain-Resample-Release sampling (R3) algorithm that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of R3 sampling to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of R3 sampling and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.