# Auto-differentiable Ensemble Kalman Filters

arXiv: https://arxiv.org/abs/2107.07687 · HTML: https://mlsanity.com/paper/2107.07687/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 95 references that are in the map.
- First posted on arXiv 16 Jul 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning](https://mlsanity.com/paper/1710.05741/) (2017)
- [Variational Sequential Monte Carlo](https://mlsanity.com/paper/1705.11140/) (2017)
- [Filtering Variational Objectives](https://mlsanity.com/paper/1705.09279/) (2017)
- [Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model](https://mlsanity.com/paper/2001.01520/) (2020)
- [Auto-Encoding Sequential Monte Carlo](https://mlsanity.com/paper/1705.10306/) (2017)
- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization](https://mlsanity.com/paper/2001.06270/) (2020)
- [Learning Dynamical Systems from Partial Observations](https://mlsanity.com/paper/1902.11136/) (2019)
- [Machine Learning for Prediction with Missing Dynamics](https://mlsanity.com/paper/1910.05861/) (2019)
- [Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation](https://mlsanity.com/paper/2007.07383/) (2020)
- [Inverse Problems and Data Assimilation](https://mlsanity.com/paper/1810.06191/) (2018)
- [The Ensemble Kalman Filter: A Signal Processing Perspective](https://mlsanity.com/paper/1702.08061/) (2017)
- [Iterative Ensemble Kalman Methods: A Unified Perspective with Some New Variants](https://mlsanity.com/paper/2010.13299/) (2020)
- [EM-like Learning Chaotic Dynamics from Noisy and Partial Observations](https://mlsanity.com/paper/1903.10335/) (2019)
- [Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time models](https://mlsanity.com/paper/1704.06988/) (2017)
- [Deterministic Mean-field Ensemble Kalman Filtering](https://mlsanity.com/paper/1409.0628/) (2014)
- [Ensemble MCMC: Accelerating Pseudo-Marginal MCMC for State Space Models using the Ensemble Kalman Filter](https://mlsanity.com/paper/1906.02014/) (2019)
- [A Bayesian adaptive ensemble Kalman filter for sequential state and parameter estimation](https://mlsanity.com/paper/1611.03835/) (2016)
- [Estimating model evidence using ensemble-based data assimilation with localization - The model selection problem](https://mlsanity.com/paper/1709.06635/) (2017)

## Built on by (most cited first, 28 of 28)

- [Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review](https://mlsanity.com/paper/2303.10462/) (2023)
- [Training neural operators to preserve invariant measures of chaotic attractors](https://mlsanity.com/paper/2306.01187/) (2023)
- [Deep Generative Data Assimilation in Multimodal Setting](https://mlsanity.com/paper/2404.06665/) (2024)
- [Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effects](https://mlsanity.com/paper/2203.05164/) (2022)
- [Reduced-Order Autodifferentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2301.11961/) (2023)
- [Learning About Structural Errors in Models of Complex Dynamical Systems](https://mlsanity.com/paper/2401.00035/) (2023)
- [Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Response](https://mlsanity.com/paper/2402.17233/) (2024)
- [Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversion](https://mlsanity.com/paper/2307.07882/) (2023)
- [Learning Optimal Filters Using Variational Inference](https://mlsanity.com/paper/2406.18066/) (2024)
- [Non-Asymptotic Analysis of Ensemble Kalman Updates: Effective Dimension and Localization](https://mlsanity.com/paper/2208.03246/) (2022)
- [Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble](https://mlsanity.com/paper/2408.04739/) (2024)
- [CGNSDE: Conditional Gaussian Neural Stochastic Differential Equation for Modeling Complex Systems and Data Assimilation](https://mlsanity.com/paper/2404.06749/) (2024)
- [Deep Bayesian Filter for Bayes-faithful Data Assimilation](https://mlsanity.com/paper/2405.18674/) (2024)
- [Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network](https://mlsanity.com/paper/2507.08749/) (2025)
- [Phi-DVAE: Physics-Informed Dynamical Variational Autoencoders for Unstructured Data Assimilation](https://mlsanity.com/paper/2209.15609/) (2022)
- [Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inference](https://mlsanity.com/paper/2312.05910/) (2023)
- [Regularization-Based Efficient Continual Learning in Deep State-Space Models](https://mlsanity.com/paper/2403.10123/) (2024)
- [Learning Absorption Rates in Glucose-Insulin Dynamics from Meal Covariates](https://mlsanity.com/paper/2304.14300/) (2023)
- [iPINNER: An Iterative Physics-Informed Neural Network with Ensemble Kalman Filter](https://mlsanity.com/paper/2506.00731/) (2025)
- [Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems](https://mlsanity.com/paper/2503.18309/) (2025)
- [Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approach](https://mlsanity.com/paper/2504.20319/) (2025)
- [Long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systems](https://mlsanity.com/paper/2412.14318/) (2024)
- [Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms](https://mlsanity.com/paper/2603.20891/) (2026)
- [Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient](https://mlsanity.com/paper/2409.15393/) (2024)
- [The Ensemble Kalman Update is an Empirical Matheron Update](https://mlsanity.com/paper/2502.03048/) (2025)
- [pDANSE: Particle-based Data-driven Nonlinear State Estimation from Nonlinear Measurements](https://mlsanity.com/paper/2510.27503/) (2025)
- [Bayesian Experimental Design for Model Discrepancy Calibration: A Rivalry between Kullback--Leibler Divergence and Wasserstein Distance](https://mlsanity.com/paper/2601.16425/) (2026)
- [Generative Model Proposal based Particle Filtering for Data Assimilation](https://mlsanity.com/paper/2607.01012/) (2026)

## Abstract (arXiv)

Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.