# Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks

arXiv: https://arxiv.org/abs/2407.01613 · HTML: https://mlsanity.com/paper/2407.01613/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Physics-Informed Deep Neural Operator Networks](https://mlsanity.com/paper/2207.05748/) (2022)
- [A physics-informed variational DeepONet for predicting the crack path in brittle materials](https://mlsanity.com/paper/2108.06905/) (2021)
- [Improved architectures and training algorithms for deep operator networks](https://mlsanity.com/paper/2110.01654/) (2021)
- [Residual-based attention in physics-informed neural networks](https://mlsanity.com/paper/2307.00379/) (2023)
- [DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equations](https://mlsanity.com/paper/2112.14038/) (2021)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems](https://mlsanity.com/paper/2204.09157/) (2022)
- [A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture](https://mlsanity.com/paper/2005.00615/) (2020)
- [Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusion](https://mlsanity.com/paper/2109.14860/) (2021)
- [Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs)](https://mlsanity.com/paper/2205.01059/) (2022)
- [Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)](https://mlsanity.com/paper/2209.09988/) (2022)
- [A multifidelity approach to continual learning for physical systems](https://mlsanity.com/paper/2304.03894/) (2023)
- [Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations](https://mlsanity.com/paper/2303.11577/) (2023)

## Built on by (most cited first, 20 of 20)

- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Inferring turbulent velocity and temperature fields and their statistics from Lagrangian velocity measurements using physics-informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.15727/) (2024)
- [SPIKANs: Separable Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2411.06286/) (2024)
- [Predicting Crack Nucleation and Propagation in Brittle Materials Using Deep Operator Networks with Diverse Trunk Architectures](https://mlsanity.com/paper/2501.00016/) (2024)
- [ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2512.03476/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [Self-adaptive weighting and sampling for physics-informed neural networks](https://mlsanity.com/paper/2511.05452/) (2025)
- [E-PINNs: Epistemic Physics-Informed Neural Networks](https://mlsanity.com/paper/2503.19333/) (2025)
- [Physics-Informed Machine Learning in Biomedical Science and Engineering](https://mlsanity.com/paper/2510.05433/) (2025)
- [Learning where to learn: Training data distribution optimization for scientific machine learning](https://mlsanity.com/paper/2505.21626/) (2025)
- [LNN-PINN: A Unified Physics-Only Training Framework with Liquid Residual Blocks](https://mlsanity.com/paper/2508.08935/) (2025)
- [Training Deep Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2510.23501/) (2025)
- [On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime](https://mlsanity.com/paper/2601.03162/) (2026)
- [Estimating Parameter Fields in Multi-Physics PDEs from Scarce Measurements](https://mlsanity.com/paper/2509.00203/) (2025)
- [Disentangling Aleatoric and Epistemic Uncertainty in Physics-Informed Neural Networks. Application to Insulation Material Degradation Prognostics](https://mlsanity.com/paper/2601.03673/) (2026)
- [Improving the accuracy of physics-informed neural networks via last-layer retraining](https://mlsanity.com/paper/2603.04672/) (2026)
- [Residual Attention Physics-Informed Neural Networks for Robust Multiphysics Simulation of Steady-State Electrothermal Energy Systems](https://mlsanity.com/paper/2603.23578/) (2026)
- [Mitigating Gradient Pathology in PINNs through Aligned Constraint](https://mlsanity.com/paper/2605.25001/) (2026)
- [Curvature-aware dynamic precision approach for physics-informed neural networks](https://mlsanity.com/paper/2606.04736/) (2026)
- [Learning Unsteady Aneurysm Hemodynamics with Physics-Informed DeepONets](https://mlsanity.com/paper/2608.13629/) (2026)

## Abstract (arXiv)

Physics-informed deep learning has emerged as a promising alternative for solving partial differential equations. However, for complex problems, training these networks can still be challenging, often resulting in unsatisfactory accuracy and efficiency. In this work, we demonstrate that the failure of plain physics-informed neural networks arises from the significant discrepancy in the convergence rate of residuals at different training points, where the slowest convergence rate dominates the overall solution convergence. Based on these observations, we propose a pointwise adaptive weighting method that balances the residual decay rate across different training points. The performance of our proposed adaptive weighting method is compared with current state-of-the-art adaptive weighting methods on benchmark problems for both physics-informed neural networks and physics-informed deep operator networks. Through extensive numerical results we demonstrate that our proposed approach of balanced residual decay rates offers several advantages, including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.