# A Framework for Machine Learning of Model Error in Dynamical Systems

arXiv: https://arxiv.org/abs/2107.06658 · HTML: https://mlsanity.com/paper/2107.06658/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 198 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 30 of 66)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Latent ODEs for Irregularly-Sampled Time Series](https://mlsanity.com/paper/1907.03907/) (2019)
- [Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems](https://mlsanity.com/paper/2003.04919/) (2020)
- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [Deep learning to represent sub-grid processes in climate models](https://mlsanity.com/paper/1806.04731/) (2018)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Solving and Learning Nonlinear PDEs with Gaussian Processes](https://mlsanity.com/paper/2103.12959/) (2021)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting](https://mlsanity.com/paper/2010.04456/) (2020)
- [The Random Feature Model for Input-Output Maps between Banach Spaces](https://mlsanity.com/paper/2005.10224/) (2020)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Neural Lander: Stable Drone Landing Control using Learned Dynamics](https://mlsanity.com/paper/1811.08027/) (2018)
- [A Priori Estimates of the Population Risk for Two-layer Neural Networks](https://mlsanity.com/paper/1810.06397/) (2018)
- [Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model](https://mlsanity.com/paper/2001.01520/) (2020)
- [Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles](https://mlsanity.com/paper/2001.11086/) (2020)
- [Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model](https://mlsanity.com/paper/1803.04779/) (2018)
- [Extracting Sparse High-Dimensional Dynamics from Limited Data](https://mlsanity.com/paper/1707.08528/) (2017)
- [EikoNet: Solving the Eikonal equation with Deep Neural Networks](https://mlsanity.com/paper/2004.00361/) (2020)
- [Integrating Expert ODEs into Neural ODEs: Pharmacology and Disease Progression](https://mlsanity.com/paper/2106.02875/) (2021)

## Built on by (most cited first, 30 of 38)

- [Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review](https://mlsanity.com/paper/2303.10462/) (2023)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations](https://mlsanity.com/paper/2108.03561/) (2021)
- [Physics-enhanced deep surrogates for partial differential equations](https://mlsanity.com/paper/2111.05841/) (2021)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Robust Hybrid Learning With Expert Augmentation](https://mlsanity.com/paper/2202.03881/) (2022)
- [Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effects](https://mlsanity.com/paper/2203.05164/) (2022)
- [Online model error correction with neural networks in the incremental 4D-Var framework](https://mlsanity.com/paper/2210.13817/) (2022)
- [Reduced-Order Autodifferentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2301.11961/) (2023)
- [Learning About Structural Errors in Models of Complex Dynamical Systems](https://mlsanity.com/paper/2401.00035/) (2023)
- [Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Response](https://mlsanity.com/paper/2402.17233/) (2024)
- [Catch-22s of reservoir computing](https://mlsanity.com/paper/2210.10211/) (2022)
- [Learning Enhanced Ensemble Filters](https://mlsanity.com/paper/2504.17836/) (2025)
- [Learning Optimal Filters Using Variational Inference](https://mlsanity.com/paper/2406.18066/) (2024)
- [Learning dynamical systems with hit-and-run random feature maps](https://mlsanity.com/paper/2501.06661/) (2025)
- [A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacoustic](https://mlsanity.com/paper/2204.06362/) (2022)
- [Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case](https://mlsanity.com/paper/2110.00546/) (2021)
- [Data assimilation and discrepancy modeling with shallow recurrent decoders](https://mlsanity.com/paper/2512.01170/) (2025)
- [How more data can hurt: Instability and regularization in next-generation reservoir computing](https://mlsanity.com/paper/2407.08641/) (2024)
- [Learning Semilinear Neural Operators : A Unified Recursive Framework For Prediction And Data Assimilation](https://mlsanity.com/paper/2402.15656/) (2024)
- [Machine Learning Applications to Computational Plasma Physics and Reduced-Order Plasma Modeling: A Perspective](https://mlsanity.com/paper/2409.02349/) (2024)
- [Learning Absorption Rates in Glucose-Insulin Dynamics from Meal Covariates](https://mlsanity.com/paper/2304.14300/) (2023)
- [Learning solution operator of dynamical systems with diffusion maps kernel ridge regression](https://mlsanity.com/paper/2512.17203/) (2025)
- [Closed-form conditional diffusion models for data assimilation](https://mlsanity.com/paper/2603.21291/) (2026)
- [Reservoir Computing with Error Correction: Long-term Behaviors of Stochastic Dynamical Systems](https://mlsanity.com/paper/2305.00669/) (2023)
- [Hybrid Modeling Design Patterns](https://mlsanity.com/paper/2401.00033/) (2023)
- [Development of an offline and online hybrid model for the Integrated Forecasting System](https://mlsanity.com/paper/2403.03702/) (2024)
- [On the choice of the non-trainable internal weights in random feature maps](https://mlsanity.com/paper/2408.03626/) (2024)
- [Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approach](https://mlsanity.com/paper/2504.20319/) (2025)
- [Structural Constraints for Physics-augmented Learning](https://mlsanity.com/paper/2410.05507/) (2024)

## Abstract (arXiv)

The development of data-informed predictive models for dynamical systems is of widespread interest in many disciplines. We present a unifying framework for blending mechanistic and machine-learning approaches to identify dynamical systems from noisily and partially observed data. We compare pure data-driven learning with hybrid models which incorporate imperfect domain knowledge. Our formulation is agnostic to the chosen machine learning model, is presented in both continuous- and discrete-time settings, and is compatible both with model errors that exhibit substantial memory and errors that are memoryless. First, we study memoryless linear (w.r.t. parametric-dependence) model error from a learning theory perspective, defining excess risk and generalization error. For ergodic continuous-time systems, we prove that both excess risk and generalization error are bounded above by terms that diminish with the square-root of T, the time-interval over which training data is specified. Secondly, we study scenarios that benefit from modeling with memory, proving universal approximation theorems for two classes of continuous-time recurrent neural networks (RNNs): both can learn memory-dependent model error. In addition, we connect one class of RNNs to reservoir computing, thereby relating learning of memory-dependent error to recent work on supervised learning between Banach spaces using random features. Numerical results are presented (Lorenz '63, Lorenz '96 Multiscale systems) to compare purely data-driven and hybrid approaches, finding hybrid methods less data-hungry and more parametrically efficient. Finally, we demonstrate numerically how data assimilation can be leveraged to learn hidden dynamics from noisy, partially-observed data, and illustrate challenges in representing memory by this approach, and in the training of such models.