# Deep Generative Data Assimilation in Multimodal Setting

arXiv: https://arxiv.org/abs/2404.06665 · HTML: https://mlsanity.com/paper/2404.06665/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Fast Sampling of Diffusion Models with Exponential Integrator](https://mlsanity.com/paper/2204.13902/) (2022)
- [Conditional Image Generation with Score-Based Diffusion Models](https://mlsanity.com/paper/2111.13606/) (2021)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [Score-based Data Assimilation](https://mlsanity.com/paper/2306.10574/) (2023)
- [On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2304.06798/) (2023)
- [DiffDA: a Diffusion Model for Weather-scale Data Assimilation](https://mlsanity.com/paper/2401.05932/) (2024)
- [ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction](https://mlsanity.com/paper/2402.00712/) (2024)
- [Combining data assimilation and machine learning to infer unresolved scale parametrisation](https://mlsanity.com/paper/2009.04318/) (2020)
- [Latent Space Data Assimilation by using Deep Learning](https://mlsanity.com/paper/2104.00430/) (2021)
- [Auto-differentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2107.07687/) (2021)
- [Position, Padding and Predictions: A Deeper Look at Position Information in CNNs](https://mlsanity.com/paper/2101.12322/) (2021)
- [Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Models](https://mlsanity.com/paper/2204.03497/) (2022)
- [Online model error correction with neural networks in the incremental 4D-Var framework](https://mlsanity.com/paper/2210.13817/) (2022)
- [Joint Parameter and Parameterization Inference with Uncertainty Quantification through Differentiable Programming](https://mlsanity.com/paper/2403.02215/) (2024)
- [History-Based, Bayesian, Closure for Stochastic Parameterization: Application to Lorenz '96](https://mlsanity.com/paper/2210.14488/) (2022)

## Built on by (most cited first, 29 of 29)

- [On conditional diffusion models for PDE simulations](https://mlsanity.com/paper/2410.16415/) (2024)
- [Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling](https://mlsanity.com/paper/2408.10958/) (2024)
- [Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales](https://mlsanity.com/paper/2406.16947/) (2024)
- [FlowDAS: A Stochastic Interpolant-based Framework for Data Assimilation](https://mlsanity.com/paper/2501.16642/) (2025)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [PhyDA: Physics-Guided Diffusion Models for Data Assimilation in Atmospheric Systems](https://mlsanity.com/paper/2505.12882/) (2025)
- [FNP: Fourier Neural Processes for Arbitrary-Resolution Data Assimilation](https://mlsanity.com/paper/2406.01645/) (2024)
- [Generative emulation of chaotic dynamics with coherent prior](https://mlsanity.com/paper/2504.14264/) (2025)
- [A Multimodal Physics-Informed Neural Network Approach for Mean Radiant Temperature Modeling](https://mlsanity.com/paper/2503.08482/) (2025)
- [Strictly Constrained Generative Modeling via Split Augmented Langevin Sampling](https://mlsanity.com/paper/2505.18017/) (2025)
- [Ensemble Kalman filter in latent space using a variational autoencoder pair](https://mlsanity.com/paper/2502.12987/) (2025)
- [Align-DA: Align Score-based Atmospheric Data Assimilation with Multiple Preferences](https://mlsanity.com/paper/2505.22008/) (2025)
- [How to use score-based diffusion in earth system science: A satellite nowcasting example](https://mlsanity.com/paper/2505.10432/) (2025)
- [Guided Unconditional and Conditional Generative Models for Super-Resolution and Inference of Quasi-Geostrophic Turbulence](https://mlsanity.com/paper/2507.00719/) (2025)
- [PIRF: Physics-Informed Reward Fine-Tuning for Diffusion Models](https://mlsanity.com/paper/2509.20570/) (2025)
- [Control-Augmented Autoregressive Diffusion for Data Assimilation](https://mlsanity.com/paper/2510.06637/) (2025)
- [DAMBench: A Multi-Modal Benchmark for Deep Learning-based Atmospheric Data Assimilation](https://mlsanity.com/paper/2511.01468/) (2025)
- [Accurate and Efficient Hybrid-Ensemble Atmospheric Data Assimilation in Latent Space with Uncertainty Quantification](https://mlsanity.com/paper/2603.04395/) (2026)
- [ADDA: a Modular Framework for Representing, Simulating and Assimilating Dynamics with End-to-end Differentiability](https://mlsanity.com/paper/2608.23297/) (2026)
- [Multi-resolution Score-Based Variational Graphical Diffusion for Causal Disaster System Modeling and Inference](https://mlsanity.com/paper/2504.04015/) (2025)
- [PnP-DA: Towards Principled Plug-and-Play Integration of Variational Data Assimilation and Generative Models](https://mlsanity.com/paper/2508.00325/) (2025)
- [Differentiable Adaptive Kalman Filtering via Optimal Transport](https://mlsanity.com/paper/2508.07037/) (2025)
- [Machine Learning Workflows in Climate Modeling: Design Patterns and Insights from Case Studies](https://mlsanity.com/paper/2510.03305/) (2025)
- [Prequential posteriors](https://mlsanity.com/paper/2511.17721/) (2025)
- [FlowDA: Accurate, Low-Latency Weather Data Assimilation via Flow Matching](https://mlsanity.com/paper/2602.06800/) (2026)
- [Uncertainty-Calibrated Spatiotemporal Field Diffusion with Sparse Supervision](https://mlsanity.com/paper/2603.04431/) (2026)
- [Generative Atmospheric Super-Resolution from Heterogeneous In Situ Observations through Composable Interfaces](https://mlsanity.com/paper/2609.29027/) (2026)
- [Benchmarking Generative Models for Weather Data Assimilation on Real Station Observations](https://mlsanity.com/paper/2610.00728/) (2026)
- [A Unified Framework for Bayesian Data Assimilation with Generative Models and Observation Interpolants](https://mlsanity.com/paper/2610.03396/) (2026)

## Abstract (arXiv)

Robust integration of physical knowledge and data is key to improve computational simulations, such as Earth system models. Data assimilation is crucial for achieving this goal because it provides a systematic framework to calibrate model outputs with observations, which can include remote sensing imagery and ground station measurements, with uncertainty quantification. Conventional methods, including Kalman filters and variational approaches, inherently rely on simplifying linear and Gaussian assumptions, and can be computationally expensive. Nevertheless, with the rapid adoption of data-driven methods in many areas of computational sciences, we see the potential of emulating traditional data assimilation with deep learning, especially generative models. In particular, the diffusion-based probabilistic framework has large overlaps with data assimilation principles: both allows for conditional generation of samples with a Bayesian inverse framework. These models have shown remarkable success in text-conditioned image generation or image-controlled video synthesis. Likewise, one can frame data assimilation as observation-conditioned state calibration. In this work, we propose SLAMS: Score-based Latent Assimilation in Multimodal Setting. Specifically, we assimilate in-situ weather station data and ex-situ satellite imagery to calibrate the vertical temperature profiles, globally. Through extensive ablation, we demonstrate that SLAMS is robust even in low-resolution, noisy, and sparse data settings. To our knowledge, our work is the first to apply deep generative framework for multimodal data assimilation using real-world datasets; an important step for building robust computational simulators, including the next-generation Earth system models. Our code is available at: https://github.com/yongquan-qu/SLAMS