# Adaptive Self-supervision Algorithms for Physics-informed Neural Networks

arXiv: https://arxiv.org/abs/2207.04084 · HTML: https://mlsanity.com/paper/2207.04084/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data](https://mlsanity.com/paper/1901.06314/) (2019)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks](https://mlsanity.com/paper/1906.05747/) (2019)
- [Multi-Objective Loss Balancing for Physics-Informed Deep Learning](https://mlsanity.com/paper/2110.09813/) (2021)
- [A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture](https://mlsanity.com/paper/2005.00615/) (2020)
- [SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs](https://mlsanity.com/paper/2102.13037/) (2021)

## Built on by (most cited first, 24 of 24)

- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [Learning Physical Models that Can Respect Conservation Laws](https://mlsanity.com/paper/2302.11002/) (2023)
- [A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions](https://mlsanity.com/paper/2302.14227/) (2023)
- [Failure-informed adaptive sampling for PINNs, Part II: combining with re-sampling and subset simulation](https://mlsanity.com/paper/2302.01529/) (2023)
- [The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operators](https://mlsanity.com/paper/2510.06646/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers](https://mlsanity.com/paper/2602.13498/) (2026)
- [Deep adaptive sampling for surrogate modeling without labeled data](https://mlsanity.com/paper/2402.11283/) (2024)
- [Deep Tree Tensor Networks](https://mlsanity.com/paper/2502.09928/) (2025)
- [Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programming](https://mlsanity.com/paper/2409.10777/) (2024)
- [Investigating Guiding Information for Adaptive Collocation Point Sampling in PINNs](https://mlsanity.com/paper/2404.12282/) (2024)
- [Optimal time sampling in physics-informed neural networks](https://mlsanity.com/paper/2404.18780/) (2024)
- [Physics-aware deep learning framework for linear elasticity](https://mlsanity.com/paper/2302.09668/) (2023)
- [Kolmogorov n-Widths for Multitask Physics-Informed Machine Learning (PIML) Methods: Towards Robust Metrics](https://mlsanity.com/paper/2402.11126/) (2024)
- [Efficient Training of Physics-Informed Neural Networks with Direct Grid Refinement Algorithm](https://mlsanity.com/paper/2306.08293/) (2023)
- [Bayesian Reasoning for Physics Informed Neural Networks](https://mlsanity.com/paper/2308.13222/) (2023)
- [Provably Accurate Adaptive Sampling for Collocation Points in Physics-informed Neural Networks](https://mlsanity.com/paper/2504.00910/) (2025)
- [M'ethode de quadrature pour les PINNs fond'ee th'eoriquement sur la hessienne des r'esiduels](https://mlsanity.com/paper/2506.20441/) (2025)
- [Estimating Parameter Fields in Multi-Physics PDEs from Scarce Measurements](https://mlsanity.com/paper/2509.00203/) (2025)
- [Auto-Adaptive PINNs with Applications to Phase Transitions](https://mlsanity.com/paper/2510.23999/) (2025)
- [Efficient Global-Local Fusion Sampling for Physics-Informed Neural Networks](https://mlsanity.com/paper/2510.24026/) (2025)
- [KD-PINN: Knowledge-Distilled PINNs for ultra-low-latency real-time neural PDE solvers](https://mlsanity.com/paper/2512.13336/) (2025)
- [Adaptive Physics-Informed Neural Networks for the Blasius Boundary-Layer Problem](https://mlsanity.com/paper/2609.22185/) (2026)

## Abstract (arXiv)

Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function, but recent work has shown that this can lead to optimization difficulties. Here, we study the impact of the location of the collocation points on the trainability of these models. We find that the vanilla PINN performance can be significantly boosted by adapting the location of the collocation points as training proceeds. Specifically, we propose a novel adaptive collocation scheme which progressively allocates more collocation points (without increasing their number) to areas where the model is making higher errors (based on the gradient of the loss function in the domain). This, coupled with a judicious restarting of the training during any optimization stalls (by simply resampling the collocation points in order to adjust the loss landscape) leads to better estimates for the prediction error. We present results for several problems, including a 2D Poisson and diffusion-advection system with different forcing functions. We find that training vanilla PINNs for these problems can result in up to 70% prediction error in the solution, especially in the regime of low collocation points. In contrast, our adaptive schemes can achieve up to an order of magnitude smaller error, with similar computational complexity as the baseline. Furthermore, we find that the adaptive methods consistently perform on-par or slightly better than vanilla PINN method, even for large collocation point regimes. The code for all the experiments has been open sourced.