# Stacked networks improve physics-informed training: applications to neural networks and deep operator networks

arXiv: https://arxiv.org/abs/2311.06483 · HTML: https://mlsanity.com/paper/2311.06483/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 80 references that are in the map.
- First posted on arXiv 11 Nov 2023.

## Builds on (most cited first, 30 of 47)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction](https://mlsanity.com/paper/2211.14680/) (2022)
- [Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations](https://mlsanity.com/paper/2107.07871/) (2021)
- [Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks](https://mlsanity.com/paper/2007.04542/) (2020)
- [Efficient training of physics-informed neural networks via importance sampling](https://mlsanity.com/paper/2104.12325/) (2021)
- [A physics-informed variational DeepONet for predicting the crack path in brittle materials](https://mlsanity.com/paper/2108.06905/) (2021)
- [Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transport](https://mlsanity.com/paper/2204.06684/) (2022)
- [Long-time integration of parametric evolution equations with physics-informed DeepONets](https://mlsanity.com/paper/2106.05384/) (2021)
- [Improved architectures and training algorithms for deep operator networks](https://mlsanity.com/paper/2110.01654/) (2021)
- [Deep transfer operator learning for partial differential equations under conditional shift](https://mlsanity.com/paper/2204.09810/) (2022)
- [Physics-Informed Neural Networks for Power Systems](https://mlsanity.com/paper/1911.03737/) (2019)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems](https://mlsanity.com/paper/2204.09157/) (2022)
- [Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems](https://mlsanity.com/paper/2203.00003/) (2022)
- [A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions](https://mlsanity.com/paper/2302.14227/) (2023)

## Built on by (most cited first, 21 of 21)

- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Finite basis Kolmogorov-Arnold networks: domain decomposition for data-driven and physics-informed problems](https://mlsanity.com/paper/2406.19662/) (2024)
- [Machine learning and domain decomposition methods -- a survey](https://mlsanity.com/paper/2312.14050/) (2023)
- [SPIKANs: Separable Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2411.06286/) (2024)
- [Ensemble and Mixture-of-Experts DeepONets For Operator Learning](https://mlsanity.com/paper/2405.11907/) (2024)
- [Multifidelity Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2410.14764/) (2024)
- [Multifidelity domain decomposition-based physics-informed neural networks and operators for time-dependent problems](https://mlsanity.com/paper/2401.07888/) (2024)
- [Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers](https://mlsanity.com/paper/2410.06308/) (2024)
- [Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features](https://mlsanity.com/paper/2510.19399/) (2025)
- [What do physics-informed DeepONets learn? Understanding and improving training for scientific computing applications](https://mlsanity.com/paper/2411.18459/) (2024)
- [mLaSDI: Multi-stage latent space dynamics identification](https://mlsanity.com/paper/2506.09207/) (2025)
- [Sequential decoder training for improved latent space dynamics identification](https://mlsanity.com/paper/2510.03535/) (2025)
- [Deep Learning without Global Optimization by Random Fourier Neural Networks](https://mlsanity.com/paper/2407.11894/) (2024)
- [Deep operator network models for predicting post-burn contraction](https://mlsanity.com/paper/2411.14555/) (2024)
- [Bridging quantum and classical computing for partial differential equations through multifidelity machine learning](https://mlsanity.com/paper/2512.05241/) (2025)
- [Two-scale Neural Networks for Partial Differential Equations with Small Parameters](https://mlsanity.com/paper/2402.17232/) (2024)
- [Multi-Fidelity Physics-Informed Neural Networks with Bayesian Uncertainty Quantification and Adaptive Residual Learning for Efficient Solution of Parametric Partial Differential Equations](https://mlsanity.com/paper/2602.01176/) (2026)
- [Curriculum-Learned Vanishing Stacked Residual PINNs for Hyperbolic PDE State Reconstruction](https://mlsanity.com/paper/2602.06996/) (2026)
- [Scientific Knowledge-Guided Machine Learning for Vessel Power Prediction: A Comparative Study](https://mlsanity.com/paper/2602.18403/) (2026)
- [Improving the accuracy of physics-informed neural networks via last-layer retraining](https://mlsanity.com/paper/2603.04672/) (2026)
- [Staged Depth Training: A Representation Curriculum for PINNs](https://mlsanity.com/paper/2609.30299/) (2026)

## Abstract (arXiv)

Physics-informed neural networks and operator networks have shown promise for effectively solving equations modeling physical systems. However, these networks can be difficult or impossible to train accurately for some systems of equations. We present a novel multifidelity framework for stacking physics-informed neural networks and operator networks that facilitates training. We successively build a chain of networks, where the output at one step can act as a low-fidelity input for training the next step, gradually increasing the expressivity of the learned model. The equations imposed at each step of the iterative process can be the same or different (akin to simulated annealing). The iterative (stacking) nature of the proposed method allows us to progressively learn features of a solution that are hard to learn directly. Through benchmark problems including a nonlinear pendulum, the wave equation, and the viscous Burgers equation, we show how stacking can be used to improve the accuracy and reduce the required size of physics-informed neural networks and operator networks.