# Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learning

arXiv: https://arxiv.org/abs/2407.07375 · HTML: https://mlsanity.com/paper/2407.07375/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [Improved architectures and training algorithms for deep operator networks](https://mlsanity.com/paper/2110.01654/) (2021)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [Separable Physics-Informed Neural Networks](https://mlsanity.com/paper/2306.15969/) (2023)
- [Highway and Residual Networks learn Unrolled Iterative Estimation](https://mlsanity.com/paper/1612.07771/) (2016)
- [Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems](https://mlsanity.com/paper/2212.03130/) (2022)
- [Recipes for when Physics Fails: Recovering Robust Learning of Physics Informed Neural Networks](https://mlsanity.com/paper/2110.13330/) (2021)
- [PyDEns: a Python Framework for Solving Differential Equations with Neural Networks](https://mlsanity.com/paper/1909.11544/) (2019)
- [Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach](https://mlsanity.com/paper/2302.13143/) (2023)
- [Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problems](https://mlsanity.com/paper/2310.15690/) (2023)

## Built on by (most cited first, 4 of 4)

- [A Practitioner's Guide to Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2510.25781/) (2025)
- [ADAPT: Lightweight, Long-Range Machine Learning Force Fields Without Graphs](https://mlsanity.com/paper/2509.24115/) (2025)
- [Partition-of-Unity Gaussian Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2604.23599/) (2026)
- [Adaptive RBF-KAN: A Comparative Evaluation of Dynamic Shape Parameters in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2605.21534/) (2026)

## Abstract (arXiv)

Background: Deep learning techniques, particularly neural networks, have revolutionized computational physics, offering powerful tools for solving complex partial differential equations (PDEs). However, ensuring stability and efficiency remains a challenge, especially in scenarios involving nonlinear and time-dependent equations. Methodology: This paper introduces novel residual-based architectures, namely the Simple Highway Network and the Squared Residual Network, designed to enhance stability and accuracy in physics-informed neural networks (PINNs). These architectures augment traditional neural networks by incorporating residual connections, which facilitate smoother weight updates and improve backpropagation efficiency. Results: Through extensive numerical experiments across various examples including linear and nonlinear, time-dependent and independent PDEs we demonstrate the efficacy of the proposed architectures. The Squared Residual Network, in particular, exhibits robust performance, achieving enhanced stability and accuracy compared to conventional neural networks. These findings underscore the potential of residual-based architectures in advancing deep learning for PDEs and computational physics applications.