# Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation

arXiv: https://arxiv.org/abs/2106.07908 · HTML: https://mlsanity.com/paper/2106.07908/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 51 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Digital Twin: Values, Challenges and Enablers](https://mlsanity.com/paper/1910.01719/) (2019)
- [Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model](https://mlsanity.com/paper/2001.01520/) (2020)
- [Coupling techniques for nonlinear ensemble filtering](https://mlsanity.com/paper/1907.00389/) (2019)
- [Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model](https://mlsanity.com/paper/1909.04711/) (2019)
- [Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization](https://mlsanity.com/paper/2001.06270/) (2020)
- [Data Assimilation: A Mathematical Introduction](https://mlsanity.com/paper/1506.07825/) (2015)
- [A Fast and Scalable Method for A-Optimal Design of Experiments for Infinite-dimensional Bayesian Nonlinear Inverse Problems](https://mlsanity.com/paper/1410.5899/) (2014)
- [Machine Learning, Deepest Learning: Statistical Data Assimilation Problems](https://mlsanity.com/paper/1707.01415/) (2017)
- [Probabilistic learning on manifolds constrained by nonlinear partial differential equations for small datasets](https://mlsanity.com/paper/2010.14324/) (2020)
- [Deterministic Mean-field Ensemble Kalman Filtering](https://mlsanity.com/paper/1409.0628/) (2014)
- [McKean-Vlasov SDEs in nonlinear filtering](https://mlsanity.com/paper/2007.12658/) (2020)
- [A-optimal encoding weights for nonlinear inverse problems, with applications to the Helmholtz inverse problem](https://mlsanity.com/paper/1612.02358/) (2016)

## Built on by (most cited first, 2 of 2)

- [Ensemble Transport Filter via Optimized Maximum Mean Discrepancy](https://mlsanity.com/paper/2407.11518/) (2024)
- [Scalable method for Bayesian experimental design without integrating over posterior distribution](https://mlsanity.com/paper/2306.17615/) (2023)

## Abstract (arXiv)

This paper presents the machine learning-based ensemble conditional mean filter (ML-EnCMF) -- a filtering method based on the conditional mean filter (CMF) previously introduced in the literature. The updated mean of the CMF matches that of the posterior, obtained by applying Bayes' rule on the filter's forecast distribution. Moreover, we show that the CMF's updated covariance coincides with the expected conditional covariance. Implementing the EnCMF requires computing the conditional mean (CM). A likelihood-based estimator is prone to significant errors for small ensemble sizes, causing the filter divergence. We develop a systematical methodology for integrating machine learning into the EnCMF based on the CM's orthogonal projection property. First, we use a combination of an artificial neural network (ANN) and a linear function, obtained based on the ensemble Kalman filter (EnKF), to approximate the CM, enabling the ML-EnCMF to inherit EnKF's advantages. Secondly, we apply a suitable variance reduction technique to reduce statistical errors when estimating loss function. Lastly, we propose a model selection procedure for element-wisely selecting the applied filter, i.e., either the EnKF or ML-EnCMF, at each updating step. We demonstrate the ML-EnCMF performance using the Lorenz-63 and Lorenz-96 systems and show that the ML-EnCMF outperforms the EnKF and the likelihood-based EnCMF.