# Improved Training of Physics-Informed Neural Networks with Model Ensembles

arXiv: https://arxiv.org/abs/2204.05108 · HTML: https://mlsanity.com/paper/2204.05108/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Learning with Pseudo-Ensembles](https://mlsanity.com/paper/1412.4864/) (2014)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Respecting causality is all you need for training physics-informed neural networks](https://mlsanity.com/paper/2203.07404/) (2022)
- [Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks](https://mlsanity.com/paper/2104.08426/) (2021)
- [Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks](https://mlsanity.com/paper/2007.04542/) (2020)
- [Parallel Physics-Informed Neural Networks via Domain Decomposition](https://mlsanity.com/paper/2104.10013/) (2021)
- [A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networks](https://mlsanity.com/paper/2007.07442/) (2020)
- [Learning in Sinusoidal Spaces with Physics-Informed Neural Networks](https://mlsanity.com/paper/2109.09338/) (2021)
- [The Old and the New: Can Physics-Informed Deep-Learning Replace Traditional Linear Solvers?](https://mlsanity.com/paper/2103.09655/) (2021)
- [Learning continuous-time PDEs from sparse data with graph neural networks](https://mlsanity.com/paper/2006.08956/) (2020)
- [A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture](https://mlsanity.com/paper/2005.00615/) (2020)
- [Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance](https://mlsanity.com/paper/1805.10407/) (2018)

## Built on by (most cited first, 19 of 19)

- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.03864/) (2024)
- [Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble](https://mlsanity.com/paper/2503.06320/) (2025)
- [Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrology](https://mlsanity.com/paper/2310.05227/) (2023)
- [On the Generalization of PINNs outside the training domain and the Hyperparameters influencing it](https://mlsanity.com/paper/2302.07557/) (2023)
- [Approximating Families of Sharp Solutions to Fisher's Equation with Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.08313/) (2024)
- [A Conformal Prediction Framework for Uncertainty Quantification in Physics-Informed Neural Networks](https://mlsanity.com/paper/2509.13717/) (2025)
- [Residual-Quantile Adjustment for Adaptive Training of Physics-informed Neural Network](https://mlsanity.com/paper/2209.05315/) (2022)
- [Symplectic Neural Flows for Modeling and Discovery](https://mlsanity.com/paper/2412.16787/) (2024)
- [PI-JEPA: Label-Free Surrogate Pretraining for Coupled Multiphysics Simulation via Operator-Split Latent Prediction](https://mlsanity.com/paper/2604.01349/) (2026)
- [Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks](https://mlsanity.com/paper/2505.17308/) (2025)
- [Label Propagation for Physics-Informed Neural Networks and Physics-Informed Gaussian Processes](https://mlsanity.com/paper/2404.05817/) (2024)
- [Hard constraint learning approaches with trainable influence functions for evolutionary equations](https://mlsanity.com/paper/2502.17497/) (2025)
- [BridgeNet: A Hybrid, Physics-Informed Machine Learning Framework for Solving High-Dimensional Fokker-Planck Equations](https://mlsanity.com/paper/2506.04354/) (2025)
- [KP-PINNs: Kernel Packet Accelerated Physics Informed Neural Networks](https://mlsanity.com/paper/2506.08563/) (2025)
- [B-PL-PINN: Stabilizing PINN Training with Bayesian Pseudo Labeling](https://mlsanity.com/paper/2507.01714/) (2025)
- [Stabilizing PINNs: A regularization scheme for PINN training to avoid unstable fixed points of dynamical systems](https://mlsanity.com/paper/2509.11768/) (2025)
- [Faster Thermal Profiling of a Lunar Rover with Machine Learning Adapted Finite Difference Model](https://mlsanity.com/paper/2605.27651/) (2026)

## Abstract (arXiv)

Learning the solution of partial differential equations (PDEs) with a neural network is an attractive alternative to traditional solvers due to its elegance, greater flexibility and the ease of incorporating observed data. However, training such physics-informed neural networks (PINNs) is notoriously difficult in practice since PINNs often converge to wrong solutions. In this paper, we address this problem by training an ensemble of PINNs. Our approach is motivated by the observation that individual PINN models find similar solutions in the vicinity of points with targets (e.g., observed data or initial conditions) while their solutions may substantially differ farther away from such points. Therefore, we propose to use the ensemble agreement as the criterion for gradual expansion of the solution interval, that is including new points for computing the loss derived from differential equations. Due to the flexibility of the domain expansion, our algorithm can easily incorporate measurements in arbitrary locations. In contrast to the existing PINN algorithms with time-adaptive strategies, the proposed algorithm does not need a pre-defined schedule of interval expansion and it treats time and space equally. We experimentally show that the proposed algorithm can stabilize PINN training and yield performance competitive to the recent variants of PINNs trained with time adaptation.