mlsanityData to 14 Oct 2026

Auto-differentiable Ensemble Kalman Filters

arXiv 2107.07687 ↗First posted 16 Jul 2021stat.ML cs.LG stat.COcited by 28

Not on the map. Why?

Abstract (arXiv)

Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 26 of its 95 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

26 of 95 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2013Black Box Variational Inferencecited by 319
  6. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 231
  7. 2016Structured Inference Networks for Nonlinear State Space Modelscited by 182
  8. 2017A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learningcited by 107
  9. 2017Variational Sequential Monte Carlocited by 95
  10. 2017Filtering Variational Objectivescited by 91
  11. 2020Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 modelcited by 74
  12. 2017Auto-Encoding Sequential Monte Carlocited by 65
14 more
  1. 2021Differentiable Particle Filtering via Entropy-Regularized Optimal Transportcited by 59
  2. 2020Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximizationcited by 38
  3. 2019Learning Dynamical Systems from Partial Observationscited by 36
  4. 2019Machine Learning for Prediction with Missing Dynamicscited by 28
  5. 2020Supervised learning from noisy observations: Combining machine-learning techniques with data assimilationcited by 27
  6. 2018Inverse Problems and Data Assimilationcited by 18
  7. 2017The Ensemble Kalman Filter: A Signal Processing Perspectivecited by 17
  8. 2020Iterative Ensemble Kalman Methods: A Unified Perspective with Some New Variantscited by 13
  9. 2019EM-like Learning Chaotic Dynamics from Noisy and Partial Observationscited by 11
  10. 2017Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time modelscited by 10
  11. 2014Deterministic Mean-field Ensemble Kalman Filteringcited by 7
  12. 2019Ensemble MCMC: Accelerating Pseudo-Marginal MCMC for State Space Models using the Ensemble Kalman Filtercited by 4
  13. 2016A Bayesian adaptive ensemble Kalman filter for sequential state and parameter estimationcited by 3
  14. 2017Estimating model evidence using ensemble-based data assimilation with localization - The model selection problemcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 9; 2025: 6; 2026: 3.

28 papers.

Sort:

  1. 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 75
  2. 2023Training neural operators to preserve invariant measures of chaotic attractorscited by 46
  3. 2024Deep Generative Data Assimilation in Multimodal Settingcited by 29
  4. 2022Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effectscited by 15
  5. 2023Reduced-Order Autodifferentiable Ensemble Kalman Filterscited by 14
  6. 2023Learning About Structural Errors in Models of Complex Dynamical Systemscited by 13
  7. 2024Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Responsecited by 13
  8. 2023Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversioncited by 9
  9. 2024Learning Optimal Filters Using Variational Inferencecited by 9
  10. 2022Non-Asymptotic Analysis of Ensemble Kalman Updates: Effective Dimension and Localizationcited by 7
  11. 2024Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemblecited by 7
  12. 2024CGNSDE: Conditional Gaussian Neural Stochastic Differential Equation for Modeling Complex Systems and Data Assimilationcited by 6
16 more
  1. 2024Deep Bayesian Filter for Bayes-faithful Data Assimilationcited by 6
  2. 2025Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Networkcited by 6
  3. 2022Phi-DVAE: Physics-Informed Dynamical Variational Autoencoders for Unstructured Data Assimilationcited by 5
  4. 2023Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inferencecited by 5
  5. 2024Regularization-Based Efficient Continual Learning in Deep State-Space Modelscited by 4
  6. 2023Learning Absorption Rates in Glucose-Insulin Dynamics from Meal Covariatescited by 3
  7. 2025iPINNER: An Iterative Physics-Informed Neural Network with Ensemble Kalman Filtercited by 3
  8. 2025Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systemscited by 2
  9. 2025Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approachcited by 2
  10. 2024Long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systemscited by 1
  11. 2026Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithmscited by 1
  12. 2024Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradientcited by 0
  13. 2025The Ensemble Kalman Update is an Empirical Matheron Updatecited by 0
  14. 2025pDANSE: Particle-based Data-driven Nonlinear State Estimation from Nonlinear Measurementscited by 0
  15. 2026Bayesian Experimental Design for Model Discrepancy Calibration: A Rivalry between Kullback--Leibler Divergence and Wasserstein Distancecited by 0
  16. 2026Generative Model Proposal based Particle Filtering for Data Assimilationcited by 0