# PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks

arXiv: https://arxiv.org/abs/2402.00326 · HTML: https://mlsanity.com/paper/2402.00326/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 89 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition](https://mlsanity.com/paper/2003.05385/) (2020)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain](https://mlsanity.com/paper/2004.13145/) (2020)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [On the Universality of Invariant Networks](https://mlsanity.com/paper/1901.09342/) (2019)
- [Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations](https://mlsanity.com/paper/2107.07871/) (2021)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks](https://mlsanity.com/paper/2007.04542/) (2020)
- [Transfer learning enhanced physics informed neural network for phase-field modeling of fracture](https://mlsanity.com/paper/1907.02531/) (2019)
- [Efficient training of physics-informed neural networks via importance sampling](https://mlsanity.com/paper/2104.12325/) (2021)

## Built on by (most cited first, 30 of 72)

- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.16371/) (2025)
- [A Practitioner's Guide to Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2510.25781/) (2025)
- [Inferring turbulent velocity and temperature fields and their statistics from Lagrangian velocity measurements using physics-informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.15727/) (2024)
- [Simulating Three-dimensional Turbulence with Physics-informed Neural Networks](https://mlsanity.com/paper/2507.08972/) (2025)
- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](https://mlsanity.com/paper/2403.18494/) (2024)
- [Deep Learning Alternatives of the Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2410.01990/) (2024)
- [FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.10949/) (2025)
- [PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations](https://mlsanity.com/paper/2412.05994/) (2024)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Initialization Schemes for Kolmogorov-Arnold Networks: An Empirical Study](https://mlsanity.com/paper/2509.03417/) (2025)
- [BiLO: Bilevel Local Operator Learning for PDE Inverse Problems](https://mlsanity.com/paper/2404.17789/) (2024)
- [Enhancing Physics-Informed Neural Networks Through Feature Engineering](https://mlsanity.com/paper/2502.07209/) (2025)
- [Learning Turbulent Flows with Generative Models: Super-resolution, Forecasting, and Sparse Flow Reconstruction](https://mlsanity.com/paper/2509.08752/) (2025)
- [Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Bases](https://mlsanity.com/paper/2410.03496/) (2024)
- [Complex Physics-Informed Neural Network](https://mlsanity.com/paper/2502.04917/) (2025)
- [ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms](https://mlsanity.com/paper/2512.03476/) (2025)
- [ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks](https://mlsanity.com/paper/2502.00803/) (2025)
- [Physics-informed Temporal Difference Metric Learning for Robot Motion Planning](https://mlsanity.com/paper/2505.05691/) (2025)
- [A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning](https://mlsanity.com/paper/2509.14198/) (2025)
- [Scale-PINN: Learning Efficient Physics-Informed Neural Networks Through Sequential Correction](https://mlsanity.com/paper/2602.19475/) (2026)
- [Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinement](https://mlsanity.com/paper/2505.18131/) (2025)
- [Physics-Informed Machine Learning in Biomedical Science and Engineering](https://mlsanity.com/paper/2510.05433/) (2025)
- [AB-PINNs: Adaptive-Basis Physics-Informed Neural Networks for Residual-Driven Domain Decomposition](https://mlsanity.com/paper/2510.08924/) (2025)
- [Element-wise Multiplication Based Deeper Physics-Informed Neural Networks](https://mlsanity.com/paper/2406.04170/) (2024)
- [Are Two Hidden Layers Still Enough for the Physics-Informed Neural Networks?](https://mlsanity.com/paper/2412.19235/) (2024)
- [EquiNO: A Physics-Informed Neural Operator for Multiscale Simulations](https://mlsanity.com/paper/2504.07976/) (2025)
- [PDE-SHARP: PDE Solver Hybrids through Analysis and Refinement Passes](https://mlsanity.com/paper/2511.00183/) (2025)
- [Enforcing hidden physics in physics-informed neural networks](https://mlsanity.com/paper/2511.14348/) (2025)

## Abstract (arXiv)

While physics-informed neural networks (PINNs) have become a popular deep learning framework for tackling forward and inverse problems governed by partial differential equations (PDEs), their performance is known to degrade when larger and deeper neural network architectures are employed. Our study identifies that the root of this counter-intuitive behavior lies in the use of multi-layer perceptron (MLP) architectures with non-suitable initialization schemes, which result in poor trainablity for the network derivatives, and ultimately lead to an unstable minimization of the PDE residual loss. To address this, we introduce Physics-informed Residual Adaptive Networks (PirateNets), a novel architecture that is designed to facilitate stable and efficient training of deep PINN models. PirateNets leverage a novel adaptive residual connection, which allows the networks to be initialized as shallow networks that progressively deepen during training. We also show that the proposed initialization scheme allows us to encode appropriate inductive biases corresponding to a given PDE system into the network architecture. We provide comprehensive empirical evidence showing that PirateNets are easier to optimize and can gain accuracy from considerably increased depth, ultimately achieving state-of-the-art results across various benchmarks. All code and data accompanying this manuscript will be made publicly available at urlhttps://github.com/PredictiveIntelligenceLab/jaxpi.