# Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged Learning

arXiv: https://arxiv.org/abs/2203.03458 · HTML: https://mlsanity.com/paper/2203.03458/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10566/) (2017)
- [Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control](https://mlsanity.com/paper/2111.10992/) (2021)
- [Data Driven Governing Equations Approximation Using Deep Neural Networks](https://mlsanity.com/paper/1811.05537/) (2018)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [Extracting Sparse High-Dimensional Dynamics from Limited Data](https://mlsanity.com/paper/1707.08528/) (2017)
- [Data-Driven Deep Learning of Partial Differential Equations in Modal Space](https://mlsanity.com/paper/1910.06948/) (2019)
- [NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data](https://mlsanity.com/paper/1908.03190/) (2019)
- [Deep Neural Network Modeling of Unknown Partial Differential Equations in Nodal Space](https://mlsanity.com/paper/2106.03603/) (2021)
- [Learning reduced systems via deep neural networks with memory](https://mlsanity.com/paper/2003.09451/) (2020)
- [Data-driven learning of non-autonomous systems](https://mlsanity.com/paper/2006.02392/) (2020)

## Built on by (most cited first, 5 of 5)

- [Flow Map Learning for Unknown Dynamical Systems: Overview, Implementation, and Benchmarks](https://mlsanity.com/paper/2307.11013/) (2023)
- [Deep Learning of Chaotic Systems from Partially-Observed Data](https://mlsanity.com/paper/2205.08384/) (2022)
- [Finite Expression Methods for Discovering Physical Laws from Data](https://mlsanity.com/paper/2305.08342/) (2023)
- [Learning Fine Scale Dynamics from Coarse Observations via Inner Recurrence](https://mlsanity.com/paper/2206.01807/) (2022)
- [Stability Preserving Data-driven Models With Latent Dynamics](https://mlsanity.com/paper/2204.11744/) (2022)

## Abstract (arXiv)

Recent work has focused on data-driven learning of the evolution of unknown systems via deep neural networks (DNNs), with the goal of conducting long time prediction of the evolution of the unknown system. Training a DNN with low generalization error is a particularly important task in this case as error is accumulated over time. Because of the inherent randomness in DNN training, chiefly in stochastic optimization, there is uncertainty in the resulting prediction, and therefore in the generalization error. Hence, the generalization error can be viewed as a random variable with some probability distribution. Well-trained DNNs, particularly those with many hyperparameters, typically result in probability distributions for generalization error with low bias but high variance. High variance causes variability and unpredictably in the results of a trained DNN. This paper presents a computational technique which decreases the variance of the generalization error, thereby improving the reliability of the DNN model to generalize consistently. In the proposed ensemble averaging method, multiple models are independently trained and model predictions are averaged at each time step. A mathematical foundation for the method is presented, including results regarding the distribution of the local truncation error. In addition, three time-dependent differential equation problems are considered as numerical examples, demonstrating the effectiveness of the method to decrease variance of DNN predictions generally.