# Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems

arXiv: https://arxiv.org/abs/2206.04811 · HTML: https://mlsanity.com/paper/2206.04811/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 61 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Towards Physics-informed Deep Learning for Turbulent Flow Prediction](https://mlsanity.com/paper/1911.08655/) (2019)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [Incorporating Symmetry into Deep Dynamics Models for Improved Generalization](https://mlsanity.com/paper/2002.03061/) (2020)
- [Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere](https://mlsanity.com/paper/2003.11927/) (2020)
- [Combining data assimilation and machine learning to infer unresolved scale parametrisation](https://mlsanity.com/paper/2009.04318/) (2020)
- [Data-Driven Aerospace Engineering: Reframing the Industry with Machine Learning](https://mlsanity.com/paper/2008.10740/) (2020)
- [Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation](https://mlsanity.com/paper/2109.12269/) (2021)
- [Long-term stability and generalization of observationally-constrained stochastic data-driven models for geophysical turbulence](https://mlsanity.com/paper/2205.04601/) (2022)
- [Machine Learning Techniques to Construct Patched Analog Ensembles for Data Assimilation](https://mlsanity.com/paper/2103.00318/) (2021)
- [Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case](https://mlsanity.com/paper/2110.00546/) (2021)
- [A multi-model ensemble Kalman filter for data assimilation and forecasting](https://mlsanity.com/paper/2202.02272/) (2022)

## Built on by (most cited first, 12 of 12)

- [Efficient deep data assimilation with sparse observations and time-varying sensors](https://mlsanity.com/paper/2310.16187/) (2023)
- [State-observation augmented diffusion model for nonlinear assimilation with unknown dynamics](https://mlsanity.com/paper/2407.21314/) (2024)
- [CGNSDE: Conditional Gaussian Neural Stochastic Differential Equation for Modeling Complex Systems and Data Assimilation](https://mlsanity.com/paper/2404.06749/) (2024)
- [BI-EqNO: Generalized Approximate Bayesian Inference with an Equivariant Neural Operator Framework](https://mlsanity.com/paper/2410.16420/) (2024)
- [Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network](https://mlsanity.com/paper/2507.08749/) (2025)
- [Ensemble Transport Filter via Optimized Maximum Mean Discrepancy](https://mlsanity.com/paper/2407.11518/) (2024)
- [RL-DAUNCE: Reinforcement Learning-Driven Data Assimilation with Uncertainty-Aware Constrained Ensembles](https://mlsanity.com/paper/2505.05452/) (2025)
- [Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1](https://mlsanity.com/paper/2407.17781/) (2024)
- [Long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systems](https://mlsanity.com/paper/2412.14318/) (2024)
- [Generating observation guided ensembles for data assimilation with denoising diffusion probabilistic model](https://mlsanity.com/paper/2308.06708/) (2023)
- [A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering](https://mlsanity.com/paper/2409.14300/) (2024)
- [Uncertainty quantification via conformal prediction in data assimilation](https://mlsanity.com/paper/2606.27001/) (2026)

## Abstract (arXiv)

Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. However, EnKF requires evolving large ensembles of forecasts using the dynamical model of the system. This often becomes computationally intractable, especially when the number of states of the system is very large, e.g., for weather prediction. With small ensembles, the estimated background error covariance matrix in the EnKF algorithm suffers from sampling error, leading to an erroneous estimate of the analysis state (initial condition for the next forecast cycle). In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case. This framework utilizes a pre-trained deep learning-based data-driven surrogate that inexpensively generates and evolves a large data-driven ensemble of the states of the system to accurately compute the background error covariance matrix with less sampling error. The H-EnKF framework estimates a better initial condition without the need for any ad-hoc localization strategies. H-EnKF can be extended to any ensemble-based DA algorithm, e.g., particle filters, which are currently difficult to use for high dimensional systems.