# Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble

arXiv: https://arxiv.org/abs/2503.06320 · HTML: https://mlsanity.com/paper/2503.06320/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 104 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 30 of 33)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism](https://mlsanity.com/paper/2009.04544/) (2020)
- [Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems](https://mlsanity.com/paper/2111.02801/) (2021)
- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Transfer learning enhanced physics informed neural network for phase-field modeling of fracture](https://mlsanity.com/paper/1907.02531/) (2019)
- [Tackling the Curse of Dimensionality with Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.12306/) (2023)
- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems](https://mlsanity.com/paper/2205.08304/) (2022)
- [Multi-stage Neural Networks: Function Approximator of Machine Precision](https://mlsanity.com/paper/2307.08934/) (2023)
- [A Physics-Guided Neural Network Framework for Elastic Plates: Comparison of Governing Equations-Based and Energy-Based Approaches](https://mlsanity.com/paper/2010.06050/) (2020)
- [NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators](https://mlsanity.com/paper/2208.11866/) (2022)
- [Correcting model misspecification in physics-informed neural networks (PINNs)](https://mlsanity.com/paper/2310.10776/) (2023)
- [L-HYDRA: Multi-Head Physics-Informed Neural Networks](https://mlsanity.com/paper/2301.02152/) (2023)
- [Discovering a reaction-diffusion model for Alzheimer's disease by combining PINNs with symbolic regression](https://mlsanity.com/paper/2307.08107/) (2023)
- [Asymptotic self-similar blow-up profile for three-dimensional axisymmetric Euler equations using neural networks](https://mlsanity.com/paper/2201.06780/) (2022)
- [Improved Training of Physics-Informed Neural Networks with Model Ensembles](https://mlsanity.com/paper/2204.05108/) (2022)
- [Physics-Informed Neural Networks for Shell Structures](https://mlsanity.com/paper/2207.14291/) (2022)
- [Leveraging viscous Hamilton-Jacobi PDEs for uncertainty quantification in scientific machine learning](https://mlsanity.com/paper/2404.08809/) (2024)
- [HomPINNs: homotopy physics-informed neural networks for solving the inverse problems of nonlinear differential equations with multiple solutions](https://mlsanity.com/paper/2304.02811/) (2023)
- [Speeding up and reducing memory usage for scientific machine learning via mixed precision](https://mlsanity.com/paper/2401.16645/) (2024)
- [NeuroSEM: A hybrid framework for simulating multiphysics problems by coupling PINNs and spectral elements](https://mlsanity.com/paper/2407.21217/) (2024)

## Built on by (most cited first, 13 of 13)

- [Physics-Informed Time-Integrated DeepONet: Temporal Tangent Space Operator Learning for High-Accuracy Inference](https://mlsanity.com/paper/2508.05190/) (2025)
- [Equivariant Flow Matching for Symmetry-Breaking Bifurcation Problems](https://mlsanity.com/paper/2509.03340/) (2025)
- [Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles](https://mlsanity.com/paper/2604.17156/) (2026)
- [Bayesian Physics Informed Neural Networks for Reliable Transformer Prognostics](https://mlsanity.com/paper/2509.15933/) (2025)
- [Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes](https://mlsanity.com/paper/2510.05568/) (2025)
- [Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks](https://mlsanity.com/paper/2505.17308/) (2025)
- [Bifurcation Models: Learning Set-Valued Solution Maps with Weight-Tied Dynamics](https://mlsanity.com/paper/2605.07277/) (2026)
- [Scalable physics-informed deep generative model for solving forward and inverse stochastic differential equations](https://mlsanity.com/paper/2503.18012/) (2025)
- [Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs](https://mlsanity.com/paper/2507.03863/) (2025)
- [Disentangling Aleatoric and Epistemic Uncertainty in Physics-Informed Neural Networks. Application to Insulation Material Degradation Prognostics](https://mlsanity.com/paper/2601.03673/) (2026)
- [Stability and Bifurcation Analysis of Nonlinear PDEs via Random Projection-based PINNs: A Krylov-Arnoldi Approach](https://mlsanity.com/paper/2603.21568/) (2026)
- [Deflation-PINNs: Learning Multiple Solutions for PDEs and Landau-de Gennes](https://mlsanity.com/paper/2603.27936/) (2026)
- [Integrating Artificial Intelligence, Physics, and Internet of Things: A Framework for Cultural Heritage Conservation](https://mlsanity.com/paper/2604.03233/) (2026)

## Abstract (arXiv)

We explore the capability of physics-informed neural networks (PINNs) to discover multiple solutions. Many real-world phenomena governed by nonlinear differential equations (DEs), such as fluid flow, exhibit multiple solutions under the same conditions, yet capturing this solution multiplicity remains a significant challenge. A key difficulty is giving appropriate initial conditions or initial guesses, to which the widely used time-marching schemes and Newton's iteration method are very sensitive in finding solutions for complex computational problems. While machine learning models, particularly PINNs, have shown promise in solving DEs, their ability to capture multiple solutions remains underexplored. In this work, we propose a simple and practical approach using PINNs to learn and discover multiple solutions. We first reveal that PINNs, when combined with random initialization and deep ensemble method -- originally developed for uncertainty quantification -- can effectively uncover multiple solutions to nonlinear ordinary and partial differential equations (ODEs/PDEs). Our approach highlights the critical role of initialization in shaping solution diversity, addressing an often-overlooked aspect of machine learning for scientific computing. Furthermore, we propose utilizing PINN-generated solutions as initial conditions or initial guesses for conventional numerical solvers to enhance accuracy and efficiency in capturing multiple solutions. Extensive numerical experiments, including the Allen-Cahn equation and cavity flow, where our approach successfully identifies both stable and unstable solutions, validate the effectiveness of our method. These findings establish a general and efficient framework for addressing solution multiplicity in nonlinear differential equations.