mlsanityData to 14 Oct 2026

LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations

arXiv 2411.19305 ↗First posted 28 Nov 2024stat.ML cs.LG math.DScited by 11

Not on the map. Why?

Abstract (arXiv)

Data assimilation techniques are crucial for accurately tracking complex dynamical systems by integrating observational data with numerical forecasts. Recently, score-based data assimilation methods emerged as powerful tools for high-dimensional and nonlinear data assimilation. However, these methods still incur substantial computational costs due to the need for expensive forward simulations. In this work, we propose LD-EnSF, a novel score-based data assimilation method that eliminates the need for full-space simulations by evolving dynamics directly in a compact latent space. Our method incorporates improved Latent Dynamics Networks (LDNets) to learn accurate surrogate dynamics and introduces a history-aware LSTM encoder to effectively process sparse and irregular observations. By operating entirely in the latent space, LD-EnSF achieves speedups orders of magnitude over existing methods while maintaining high accuracy and robustness. We demonstrate the effectiveness of LD-EnSF on several challenging high-dimensional benchmarks with highly sparse (in both space and time) and noisy observations.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 72 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

37 of 72 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  7. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  8. 2021Machine learning accelerated computational fluid dynamicscited by 471
  9. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  10. 2023Spherical Fourier Neural Operators: Learning Stable Dynamics on the Spherecited by 233
  11. 2023Score-based Data Assimilationcited by 96
  12. 2020Multiscale Simulations of Complex Systems by Learning their Effective Dynamicscited by 86
25 more
  1. 2024DiffDA: a Diffusion Model for Weather-scale Data Assimilationcited by 61
  2. 2021Data-driven discovery of intrinsic dynamicscited by 45
  3. 2023An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systemscited by 38
  4. 2021Latent Space Data Assimilation by using Deep Learningcited by 30
  5. 2022Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Modelscited by 26
  6. 2024Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Datacited by 22
  7. 2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observationscited by 21
  8. 2021Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimationcited by 21
  9. 2024Derivative-enhanced Deep Operator Networkcited by 19
  10. 2018Inverse Problems and Data Assimilationcited by 18
  11. 2023Reduced-Order Autodifferentiable Ensemble Kalman Filterscited by 14
  12. 2022Multimodal 4DVarNets for the reconstruction of sea surface dynamics from SST-SSH synergiescited by 13
  13. 20224DVarNet-SSH: end-to-end learning of variational interpolation schemes for nadir and wide-swath satellite altimetrycited by 10
  14. 2023Neural Koopman prior for data assimilationcited by 10
  15. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  16. 2023Latent assimilation with implicit neural representations for unknown dynamicscited by 9
  17. 2024State-observation augmented diffusion model for nonlinear assimilation with unknown dynamicscited by 8
  18. 2024Nonlinear Assimilation via Score-based Sequential Langevin Samplingcited by 8
  19. 2024Deep Bayesian Filter for Bayes-faithful Data Assimilationcited by 6
  20. 2024TGPT-PINN: Nonlinear model reduction with transformed GPT-PINNscited by 5
  21. 2024Learning Semilinear Neural Operators : A Unified Recursive Framework For Prediction And Data Assimilationcited by 4
  22. 2024KODA: A Data-Driven Recursive Model for Time Series Forecasting and Data Assimilation using Koopman Operatorscited by 4
  23. 2024The Deep Latent Space Particle Filter for Real-Time Data Assimilation with Uncertainty Quantificationcited by 3
  24. 2024DeltaPhi: Physical States Residual Learning for Neural Operators in Data-Limited PDE Solvingcited by 3
  25. 2024Liquid Fourier Latent Dynamics Networks for fast GPU-based numerical simulations in computational cardiologycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Flow Matching for Efficient and Scalable Data Assimilationcited by 12
  2. 2025Variationally correct operator learning: Reduced basis neural operator with a posteriori error estimationcited by 7
  3. 2026LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamicscited by 4
  4. 2026Toward AI-Driven Digital Twins for Metropolitan Floods: A Conditional Latent Dynamics Network Surrogate of the Shallow Water Equationscited by 1
  5. 2025Data-Driven Model Reduction using WeldNet: Windowed Encoders for Learning Dynamicscited by 0
  6. 2026Generative Model Proposal based Particle Filtering for Data Assimilationcited by 0
  7. 2026Ensemble Controlled-Flow Filtering for Implicit Data Assimilationcited by 0
  8. 2026TRACE: Retrospective Streaming Generation of Physical Fields under Sparse Structured Sensingcited by 0
  9. 2026AECSF: Adaptive Ensemble Conditional Score Filtering for High-Dimensional Nonlinear Data Assimilationcited by 0
  10. 2026C-STRIDE: An Observation-Driven AI Digital Twin for Predicting Basin-Wide Flood Fields from Sparse Stream-Gauge Historiescited by 0
  11. 2026LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observationscited by 0