# Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

arXiv: https://arxiv.org/abs/2001.06270 · HTML: https://mlsanity.com/paper/2001.06270/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2020.

## Built on by (most cited first, 30 of 38)

- [Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review](https://mlsanity.com/paper/2303.10462/) (2023)
- [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021)
- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [Auto-differentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2107.07687/) (2021)
- [Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations](https://mlsanity.com/paper/2108.03561/) (2021)
- [Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation](https://mlsanity.com/paper/2109.12269/) (2021)
- [Online model error correction with neural networks in the incremental 4D-Var framework](https://mlsanity.com/paper/2210.13817/) (2022)
- [Reduced-Order Autodifferentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2301.11961/) (2023)
- [A comparison of combined data assimilation and machine learning methods for offline and online model error correction](https://mlsanity.com/paper/2107.11114/) (2021)
- [Fourier neural operator for real-time simulation of 3D dynamic urban microclimate](https://mlsanity.com/paper/2308.03985/) (2023)
- [Learning stochastic dynamical systems with neural networks mimicking the Euler-Maruyama scheme](https://mlsanity.com/paper/2105.08449/) (2021)
- [State, global and local parameter estimation using local ensemble Kalman filters: applications to online machine learning of chaotic dynamics](https://mlsanity.com/paper/2107.11253/) (2021)
- [Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble](https://mlsanity.com/paper/2408.04739/) (2024)
- [A Scalable Real-Time Data Assimilation Framework for Predicting Turbulent Atmosphere Dynamics](https://mlsanity.com/paper/2407.12168/) (2024)
- [TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions](https://mlsanity.com/paper/2409.00244/) (2024)
- [Ensemble Kalman filter in latent space using a variational autoencoder pair](https://mlsanity.com/paper/2502.12987/) (2025)
- [Inversion of sea surface currents from satellite-derived SST-SSH synergies with 4DVarNets](https://mlsanity.com/paper/2211.13059/) (2022)
- [Learning Subgrid-scale Models with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2212.09967/) (2022)
- [`Next Generation' Reservoir Computing: an Empirical Data-Driven Expression of Dynamical Equations in Time-Stepping Form](https://mlsanity.com/paper/2201.05193/) (2022)
- [Learning 4DVAR inversion directly from observations](https://mlsanity.com/paper/2211.09741/) (2022)
- [Combined Optimization of Dynamics and Assimilation with End-to-End Learning on Sparse Observations](https://mlsanity.com/paper/2409.07137/) (2024)
- [Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation](https://mlsanity.com/paper/2106.07908/) (2021)
- [STEADY: Simultaneous State Estimation and Dynamics Learning from Indirect Observations](https://mlsanity.com/paper/2203.01299/) (2022)
- [Development of an offline and online hybrid model for the Integrated Forecasting System](https://mlsanity.com/paper/2403.03702/) (2024)
- [RL-DAUNCE: Reinforcement Learning-Driven Data Assimilation with Uncertainty-Aware Constrained Ensembles](https://mlsanity.com/paper/2505.05452/) (2025)
- [Ensemble forecasts in reproducing kernel Hilbert space family](https://mlsanity.com/paper/2207.14653/) (2022)
- [Long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systems](https://mlsanity.com/paper/2412.14318/) (2024)
- [Reinforcement learning based data assimilation for unknown state model](https://mlsanity.com/paper/2511.02286/) (2025)
- [Integrating Score-Based Diffusion Models with Machine Learning-Enhanced Localization for Advanced Data Assimilation in Geological Carbon Storage](https://mlsanity.com/paper/2511.05266/) (2025)
- [Balancing Accuracy and Speed: A Multi-Fidelity Ensemble Kalman Filter with a Machine Learning Surrogate Model](https://mlsanity.com/paper/2512.12276/) (2025)

## Abstract (arXiv)

The reconstruction from observations of high-dimensional chaotic dynamics such as geophysical flows is hampered by (i) the partial and noisy observations that can realistically be obtained, (ii) the need to learn from long time series of data, and (iii) the unstable nature of the dynamics. To achieve such inference from the observations over long time series, it has been suggested to combine data assimilation and machine learning in several ways. We show how to unify these approaches from a Bayesian perspective using expectation-maximization and coordinate descents. In doing so, the model, the state trajectory and model error statistics are estimated all together. Implementations and approximations of these methods are discussed. Finally, we numerically and successfully test the approach on two relevant low-order chaotic models with distinct identifiability.