# A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks

arXiv: https://arxiv.org/abs/2207.10289 · HTML: https://mlsanity.com/paper/2207.10289/

## Key facts

- Cited by 236 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 20 of 20)

- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems](https://mlsanity.com/paper/1809.08327/) (2018)
- [Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks](https://mlsanity.com/paper/2007.04542/) (2020)
- [PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs](https://mlsanity.com/paper/1909.10145/) (2019)
- [Parallel Physics-Informed Neural Networks via Domain Decomposition](https://mlsanity.com/paper/2104.10013/) (2021)
- [Efficient training of physics-informed neural networks via importance sampling](https://mlsanity.com/paper/2104.12325/) (2021)
- [A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networks](https://mlsanity.com/paper/2007.07442/) (2020)
- [Meta-learning PINN loss functions](https://mlsanity.com/paper/2107.05544/) (2021)
- [Investigating molecular transport in the human brain from MRI with physics-informed neural networks](https://mlsanity.com/paper/2205.02592/) (2022)
- [Systems Biology: Identifiability analysis and parameter identification via systems-biology informed neural networks](https://mlsanity.com/paper/2202.01723/) (2022)
- [Improved Training of Physics-Informed Neural Networks with Model Ensembles](https://mlsanity.com/paper/2204.05108/) (2022)
- [RANG: A Residual-based Adaptive Node Generation Method for Physics-Informed Neural Networks](https://mlsanity.com/paper/2205.01051/) (2022)
- [State-of-the-Art Review of Design of Experiments for Physics-Informed Deep Learning](https://mlsanity.com/paper/2202.06416/) (2022)
- [Revisiting PINNs: Generative Adversarial Physics-informed Neural Networks and Point-weighting Method](https://mlsanity.com/paper/2205.08754/) (2022)

## Built on by (most cited first, 30 of 236)

- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks](https://mlsanity.com/paper/2406.02917/) (2024)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Residual-based attention in physics-informed neural networks](https://mlsanity.com/paper/2307.00379/) (2023)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Multi-stage Neural Networks: Function Approximator of Machine Precision](https://mlsanity.com/paper/2307.08934/) (2023)
- [Fourier-MIONet: Fourier-enhanced multiple-input neural operators for multiphase modeling of geological carbon sequestration](https://mlsanity.com/paper/2303.04778/) (2023)
- [Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.16371/) (2025)
- [Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.17611/) (2024)
- [Finite basis Kolmogorov-Arnold networks: domain decomposition for data-driven and physics-informed problems](https://mlsanity.com/paper/2406.19662/) (2024)
- [PINNACLE: PINN Adaptive ColLocation and Experimental points selection](https://mlsanity.com/paper/2404.07662/) (2024)
- [Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?](https://mlsanity.com/paper/2405.04230/) (2024)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [Rethinking materials simulations: Blending direct numerical simulations with neural operators](https://mlsanity.com/paper/2312.05410/) (2023)
- [Active Learning for Neural PDE Solvers](https://mlsanity.com/paper/2408.01536/) (2024)
- [Efficient physics-informed neural networks using hash encoding](https://mlsanity.com/paper/2302.13397/) (2023)
- [Stacked networks improve physics-informed training: applications to neural networks and deep operator networks](https://mlsanity.com/paper/2311.06483/) (2023)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks](https://mlsanity.com/paper/2407.01613/) (2024)
- [Improving physics-informed neural networks with meta-learned optimization](https://mlsanity.com/paper/2303.07127/) (2023)
- [PPDONet: Deep Operator Networks for Fast Prediction of Steady-State Solutions in Disk-Planet Systems](https://mlsanity.com/paper/2305.11111/) (2023)
- [Adversarial Adaptive Sampling: Unify PINN and Optimal Transport for the Approximation of PDEs](https://mlsanity.com/paper/2305.18702/) (2023)
- [Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal](https://mlsanity.com/paper/2303.01560/) (2023)
- [An adaptive augmented Lagrangian method for training physics and equality constrained artificial neural networks](https://mlsanity.com/paper/2306.04904/) (2023)

## Abstract (arXiv)

Physics-informed neural networks (PINNs) have shown to be an effective tool for solving forward and inverse problems of partial differential equations (PDEs). PINNs embed the PDEs into the loss of the neural network, and this PDE loss is evaluated at a set of scattered residual points. The distribution of these points are highly important to the performance of PINNs. However, in the existing studies on PINNs, only a few simple residual point sampling methods have mainly been used. Here, we present a comprehensive study of two categories of sampling: non-adaptive uniform sampling and adaptive nonuniform sampling. We consider six uniform sampling, including (1) equispaced uniform grid, (2) uniformly random sampling, (3) Latin hypercube sampling, (4) Halton sequence, (5) Hammersley sequence, and (6) Sobol sequence. We also consider a resampling strategy for uniform sampling. To improve the sampling efficiency and the accuracy of PINNs, we propose two new residual-based adaptive sampling methods: residual-based adaptive distribution (RAD) and residual-based adaptive refinement with distribution (RAR-D), which dynamically improve the distribution of residual points based on the PDE residuals during training. Hence, we have considered a total of 10 different sampling methods, including six non-adaptive uniform sampling, uniform sampling with resampling, two proposed adaptive sampling, and an existing adaptive sampling. We extensively tested the performance of these sampling methods for four forward problems and two inverse problems in many setups. Our numerical results presented in this study are summarized from more than 6000 simulations of PINNs. We show that the proposed adaptive sampling methods of RAD and RAR-D significantly improve the accuracy of PINNs with fewer residual points. The results obtained in this study can also be used as a practical guideline in choosing sampling methods.