# LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations

arXiv: https://arxiv.org/abs/2610.04034 · HTML: https://mlsanity.com/paper/2610.04034/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 27 of 27)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Score-based Data Assimilation](https://mlsanity.com/paper/2306.10574/) (2023)
- [DiffDA: a Diffusion Model for Weather-scale Data Assimilation](https://mlsanity.com/paper/2401.05932/) (2024)
- [On the Guidance of Flow Matching](https://mlsanity.com/paper/2502.02150/) (2025)
- [An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems](https://mlsanity.com/paper/2309.00983/) (2023)
- [Latent Space Data Assimilation by using Deep Learning](https://mlsanity.com/paper/2104.00430/) (2021)
- [Data Assimilation: A Mathematical Introduction](https://mlsanity.com/paper/1506.07825/) (2015)
- [Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Models](https://mlsanity.com/paper/2204.03497/) (2022)
- [Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data](https://mlsanity.com/paper/2409.00127/) (2024)
- [Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations](https://mlsanity.com/paper/2108.03561/) (2021)
- [Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation](https://mlsanity.com/paper/2109.12269/) (2021)
- [FlowDAS: A Stochastic Interpolant-based Framework for Data Assimilation](https://mlsanity.com/paper/2501.16642/) (2025)
- [Reduced-Order Autodifferentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2301.11961/) (2023)
- [Flow Matching for Efficient and Scalable Data Assimilation](https://mlsanity.com/paper/2508.13313/) (2025)
- [LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations](https://mlsanity.com/paper/2411.19305/) (2024)
- [Neural Koopman prior for data assimilation](https://mlsanity.com/paper/2309.05317/) (2023)
- [Latent assimilation with implicit neural representations for unknown dynamics](https://mlsanity.com/paper/2309.09574/) (2023)
- [State-observation augmented diffusion model for nonlinear assimilation with unknown dynamics](https://mlsanity.com/paper/2407.21314/) (2024)
- [Nonlinear Assimilation via Score-based Sequential Langevin Sampling](https://mlsanity.com/paper/2411.13443/) (2024)
- [Deep Bayesian Filter for Bayes-faithful Data Assimilation](https://mlsanity.com/paper/2405.18674/) (2024)
- [KODA: A Data-Driven Recursive Model for Time Series Forecasting and Data Assimilation using Koopman Operators](https://mlsanity.com/paper/2409.19518/) (2024)
- [Tensor-Var: Efficient Four-Dimensional Variational Data Assimilation](https://mlsanity.com/paper/2501.13312/) (2025)
- [The Deep Latent Space Particle Filter for Real-Time Data Assimilation with Uncertainty Quantification](https://mlsanity.com/paper/2406.02204/) (2024)

## Abstract (arXiv)

Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.