# Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model arXiv: https://arxiv.org/abs/2001.01520 ยท HTML: https://mlsanity.com/paper/2001.01520/ ## Key facts - Cited by 74 papers in the map (data through 14 Oct 2026). - First posted on arXiv 6 Jan 2020. ## Built on by (most cited first, 30 of 74) - [Score-based Data Assimilation](https://mlsanity.com/paper/2306.10574/) (2023) - [Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review](https://mlsanity.com/paper/2303.10462/) (2023) - [ACE2: Accurately learning subseasonal to decadal atmospheric variability and forced responses](https://mlsanity.com/paper/2411.11268/) (2024) - [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021) - [Latent Space Data Assimilation by using Deep Learning](https://mlsanity.com/paper/2104.00430/) (2021) - [Variational Data Assimilation with a Learned Inverse Observation Operator](https://mlsanity.com/paper/2102.11192/) (2021) - [Auto-differentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2107.07687/) (2021) - [Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Models](https://mlsanity.com/paper/2204.03497/) (2022) - [FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation](https://mlsanity.com/paper/2312.12455/) (2023) - [Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations](https://mlsanity.com/paper/2108.03561/) (2021) - [Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation](https://mlsanity.com/paper/2109.12269/) (2021) - [FuXi Weather: A data-to-forecast machine learning system for global weather](https://mlsanity.com/paper/2408.05472/) (2024) - [Fuxi-DA: A Generalized Deep Learning Data Assimilation Framework for Assimilating Satellite Observations](https://mlsanity.com/paper/2404.08522/) (2024) - [Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers](https://mlsanity.com/paper/2103.09360/) (2021) - [Bridging observation, theory and numerical simulation of the ocean using Machine Learning](https://mlsanity.com/paper/2104.12506/) (2021) - [Online model error correction with neural networks in the incremental 4D-Var framework](https://mlsanity.com/paper/2210.13817/) (2022) - [Reduced-Order Autodifferentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2301.11961/) (2023) - [A comparison of combined data assimilation and machine learning methods for offline and online model error correction](https://mlsanity.com/paper/2107.11114/) (2021) - [Flipped Classroom: Effective Teaching for Time Series Forecasting](https://mlsanity.com/paper/2210.08959/) (2022) - [Learning to Assimilate in Chaotic Dynamical Systems](https://mlsanity.com/paper/2111.01058/) (2021) - [Heterogeneous Stream-reservoir Graph Networks with Data Assimilation](https://mlsanity.com/paper/2110.04959/) (2021) - [Deep learning delay coordinate dynamics for chaotic attractors from partial observable data](https://mlsanity.com/paper/2211.11061/) (2022) - [Reconstruction, forecasting, and stability of chaotic dynamics from partial data](https://mlsanity.com/paper/2305.15111/) (2023) - [FREE: The Foundational Semantic Recognition for Modeling Environmental Ecosystems](https://mlsanity.com/paper/2311.10255/) (2023) - [State, global and local parameter estimation using local ensemble Kalman filters: applications to online machine learning of chaotic dynamics](https://mlsanity.com/paper/2107.11253/) (2021) - [Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble](https://mlsanity.com/paper/2408.04739/) (2024) - [Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case](https://mlsanity.com/paper/2110.00546/) (2021) - [Continuous-time identification of dynamic state-space models by deep subspace encoding](https://mlsanity.com/paper/2204.09405/) (2022) - [CGNSDE: Conditional Gaussian Neural Stochastic Differential Equation for Modeling Complex Systems and Data Assimilation](https://mlsanity.com/paper/2404.06749/) (2024) - [BI-EqNO: Generalized Approximate Bayesian Inference with an Equivariant Neural Operator Framework](https://mlsanity.com/paper/2410.16420/) (2024) ## Abstract (arXiv) A novel method, based on the combination of data assimilation and machine learning is introduced. The new hybrid approach is designed for a two-fold scope: (i) emulating hidden, possibly chaotic, dynamics and (ii) predicting their future states. The method consists in applying iteratively a data assimilation step, here an ensemble Kalman filter, and a neural network. Data assimilation is used to optimally combine a surrogate model with sparse noisy data. The output analysis is spatially complete and is used as a training set by the neural network to update the surrogate model. The two steps are then repeated iteratively. Numerical experiments have been carried out using the chaotic 40-variables Lorenz 96 model, proving both convergence and statistical skill of the proposed hybrid approach. The surrogate model shows short-term forecast skill up to two Lyapunov times, the retrieval of positive Lyapunov exponents as well as the more energetic frequencies of the power density spectrum. The sensitivity of the method to critical setup parameters is also presented: the forecast skill decreases smoothly with increased observational noise but drops abruptly if less than half of the model domain is observed. The successful synergy between data assimilation and machine learning, proven here with a low-dimensional system, encourages further investigation of such hybrids with more sophisticated dynamics.