mlsanityData to 14 Oct 2026

A Framework for Machine Learning of Model Error in Dynamical Systems

arXiv 2107.06658 ↗First posted 14 Jul 2021math.DS cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

The development of data-informed predictive models for dynamical systems is of widespread interest in many disciplines. We present a unifying framework for blending mechanistic and machine-learning approaches to identify dynamical systems from noisily and partially observed data. We compare pure data-driven learning with hybrid models which incorporate imperfect domain knowledge. Our formulation is agnostic to the chosen machine learning model, is presented in both continuous- and discrete-time settings, and is compatible both with model errors that exhibit substantial memory and errors that are memoryless. First, we study memoryless linear (w.r.t. parametric-dependence) model error from a learning theory perspective, defining excess risk and generalization error. For ergodic continuous-time systems, we prove that both excess risk and generalization error are bounded above by terms that diminish with the square-root of T, the time-interval over which training data is specified. Secondly, we study scenarios that benefit from modeling with memory, proving universal approximation theorems for two classes of continuous-time recurrent neural networks (RNNs): both can learn memory-dependent model error. In addition, we connect one class of RNNs to reservoir computing, thereby relating learning of memory-dependent error to recent work on supervised learning between Banach spaces using random features. Numerical results are presented (Lorenz '63, Lorenz '96 Multiscale systems) to compare purely data-driven and hybrid approaches, finding hybrid methods less data-hungry and more parametrically efficient. Finally, we demonstrate numerically how data assimilation can be leveraged to learn hidden dynamics from noisy, partially-observed data, and illustrate challenges in representing memory by this approach, and in the training of such models.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 66 of its 198 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

66 of 198 references.

References · 66

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  6. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  7. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  8. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
  9. 2020Universal Differential Equations for Scientific Machine Learningcited by 353
  10. 2017Stable Architectures for Deep Neural Networkscited by 341
  11. 2020Model Reduction and Neural Networks for Parametric PDEscited by 258
  12. 2019Latent ODEs for Irregularly-Sampled Time Seriescited by 183
38 more
  1. 2020Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systemscited by 164
  2. 2021Next Generation Reservoir Computingcited by 160
  3. 2018Deep learning to represent sub-grid processes in climate modelscited by 156
  4. 2019Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamicscited by 134
  5. 2021Solving and Learning Nonlinear PDEs with Gaussian Processescited by 129
  6. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  7. 2020Augmenting Physical Models with Deep Networks for Complex Dynamics Forecastingcited by 95
  8. 2020The Random Feature Model for Input-Output Maps between Banach Spacescited by 90
  9. 2018Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systemscited by 89
  10. 2018Echo state networks are universalcited by 87
  11. 2018Neural Lander: Stable Drone Landing Control using Learned Dynamicscited by 85
  12. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  13. 2020Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 modelcited by 74
  14. 2020Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profilescited by 59
  15. 2018Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Modelcited by 57
  16. 2017Extracting Sparse High-Dimensional Dynamics from Limited Datacited by 50
  17. 2020EikoNet: Solving the Eikonal equation with Deep Neural Networkscited by 50
  18. 2021Integrating Expert ODEs into Neural ODEs: Pharmacology and Disease Progressioncited by 48
  19. 2014Variational Gaussian Process State-Space Modelscited by 43
  20. 2020Using machine learning to correct model error in data assimilation and forecast applicationscited by 41
  21. 2020Combining data assimilation and machine learning to infer unresolved scale parametrisationcited by 40
  22. 2020Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximizationcited by 38
  23. 2020Automatic Differentiation to Simultaneously Identify Nonlinear Dynamics and Extract Noise Probability Distributions from Datacited by 37
  24. 2019Learning Dynamical Systems from Partial Observationscited by 36
  25. 2019Machine Learning for Prediction with Missing Dynamicscited by 28
  26. 2020Supervised learning from noisy observations: Combining machine-learning techniques with data assimilationcited by 27
  27. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 27
  28. 2020Generative ODE Modeling with Known Unknownscited by 24
  29. 2020Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systemscited by 23
  30. 2020Neural Closure Models for Dynamical Systemscited by 22
  31. 2018Model Reduction with Memory and the Machine Learning of Dynamical Systemscited by 21
  32. 2017DR-RNN: A deep residual recurrent neural network for model reductioncited by 20
  33. 2019Learning Latent Dynamics for Partially-Observed Chaotic Systemscited by 18
  34. 2019Discovery of Dynamics Using Linear Multistep Methodscited by 17
  35. 2020Data-driven super-parameterization using deep learning: Experimentation with multi-scale Lorenz 96 systems and transfer-learningcited by 17
  36. 2020Reservoir Computing meets Recurrent Kernels and Structured Transformscited by 16
  37. 2019Applying machine learning to improve simulations of a chaotic dynamical system using empirical error correctioncited by 15
  38. 2019Learning Discrepancy Models From Experimental Datacited by 15

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 7; 2024: 10; 2025: 7; 2026: 6.

38 papers.

Sort:

By area:

  1. 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 75
  2. 2024Zero-shot forecasting of chaotic systemscited by 28
  3. 2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observationscited by 21
  4. 2021Physics-enhanced deep surrogates for partial differential equationscited by 18
  5. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 18
  6. 2022Robust Hybrid Learning With Expert Augmentationcited by 17
  7. 2022Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effectscited by 15
  8. 2022Online model error correction with neural networks in the incremental 4D-Var frameworkcited by 14
  9. 2023Reduced-Order Autodifferentiable Ensemble Kalman Filterscited by 14
  10. 2023Learning About Structural Errors in Models of Complex Dynamical Systemscited by 13
  11. 2024Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Responsecited by 13
  12. 2022Catch-22s of reservoir computingcited by 11
26 more
  1. 2025Learning Enhanced Ensemble Filterscited by 11
  2. 2024Learning Optimal Filters Using Variational Inferencecited by 9
  3. 2025Learning dynamical systems with hit-and-run random feature mapscited by 8
  4. 2022A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacousticcited by 7
  5. 2021Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test casecited by 6
  6. 2025Data assimilation and discrepancy modeling with shallow recurrent decoderscited by 6
  7. 2024How more data can hurt: Instability and regularization in next-generation reservoir computingcited by 5
  8. 2024Learning Semilinear Neural Operators : A Unified Recursive Framework For Prediction And Data Assimilationcited by 4
  9. 2024Machine Learning Applications to Computational Plasma Physics and Reduced-Order Plasma Modeling: A Perspectivecited by 4
  10. 2023Learning Absorption Rates in Glucose-Insulin Dynamics from Meal Covariatescited by 3
  11. 2025Learning solution operator of dynamical systems with diffusion maps kernel ridge regressioncited by 3
  12. 2026Closed-form conditional diffusion models for data assimilationcited by 3
  13. 2023Reservoir Computing with Error Correction: Long-term Behaviors of Stochastic Dynamical Systemscited by 2
  14. 2023Hybrid Modeling Design Patternscited by 2
  15. 2024Development of an offline and online hybrid model for the Integrated Forecasting Systemcited by 2
  16. 2024On the choice of the non-trainable internal weights in random feature mapscited by 2
  17. 2025Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approachcited by 2
  18. 2024Structural Constraints for Physics-augmented Learningcited by 1
  19. 2025A Weak Penalty Neural ODE for Learning Chaotic Dynamics from Noisy Time Seriescited by 1
  20. 2026Operator Learning for Smoothing and Forecastingcited by 1
  21. 2026Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithmscited by 1
  22. 2026The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modelingcited by 1
  23. 2023Interpretable Mechanistic Representations for Meal-level Glycemic Control in the Wildcited by 0
  24. 2024Pathspace Kalman Filters with Dynamic Process Uncertainty for Analyzing Time-course Datacited by 0
  25. 2026Bayesian Experimental Design for Model Discrepancy Calibration: A Rivalry between Kullback--Leibler Divergence and Wasserstein Distancecited by 0
  26. 2026Learning Closure of Dynamical Systems with Kernel Ridge Regressioncited by 0