Combining data assimilation and machine learning to infer unresolved scale parametrisation
Not on the map. Why?
Abstract (arXiv)
In recent years, machine learning (ML) has been proposed to devise data-driven parametrisations of unresolved processes in dynamical numerical models. In most cases, the ML training leverages high-resolution simulations to provide a dense, noiseless target state. Our goal is to go beyond the use of high-resolution simulations and train ML-based parametrisation using direct data, in the realistic scenario of noisy and sparse observations. The algorithm proposed in this work is a two-step process. First, data assimilation (DA) techniques are applied to estimate the full state of the system from a truncated model. The unresolved part of the truncated model is viewed as a model error in the DA system. In a second step, ML is used to emulate the unresolved part, a predictor of model error given the state of the system. Finally, the ML-based parametrisation model is added to the physical core truncated model to produce a hybrid model. The DA component of the proposed method relies on an ensemble Kalman filter while the ML parametrisation is represented by a neural network. The approach is applied to the two-scale Lorenz model and to MAOOAM, a reduced-order coupled ocean-atmosphere model. We show that in both cases the hybrid model yields forecasts with better skill than the truncated model. Moreover, the attractor of the system is significantly better represented by the hybrid model than by the truncated model.
Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Sep 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 11; 2022: 3; 2023: 7; 2024: 10; 2025: 6; 2026: 3.
40 papers.
Sort:
By area:
- 2023Score-based Data Assimilationcited by 91
- 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 74
- 2021A Framework for Machine Learning of Model Error in Dynamical Systemscited by 37
- 2021Variational Data Assimilation with a Learned Inverse Observation Operatorcited by 28
- 2024Deep Generative Data Assimilation in Multimodal Settingcited by 28
- 2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observationscited by 20
- 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 19
- 2021Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformerscited by 15
- 2021Bridging observation, theory and numerical simulation of the ocean using Machine Learningcited by 15
- 2022Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effectscited by 15
- 2021A comparison of combined data assimilation and machine learning methods for offline and online model error correctioncited by 13
- 2022Online model error correction with neural networks in the incremental 4D-Var frameworkcited by 13
28 more
- 2023Learning About Structural Errors in Models of Complex Dynamical Systemscited by 13
- 2024Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Responsecited by 13
- 2022Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systemscited by 12
- 2024Global atmospheric data assimilation with multi-modal masked autoencoderscited by 11
- 2021Super-resolution data assimilationcited by 8
- 2021Observation Error Covariance Specification in Dynamical Systems for Data assimilation using Recurrent Neural Networkscited by 7
- 2023Digital Twins in Wind Energy: Emerging Technologies and Industry-Informed Future Directionscited by 7
- 2024AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storagecited by 7
- 2024State-observation augmented diffusion model for nonlinear assimilation with unknown dynamicscited by 7
- 2024Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemblecited by 7
- 2021State, global and local parameter estimation using local ensemble Kalman filters: applications to online machine learning of chaotic dynamicscited by 6
- 2021Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test casecited by 6
- 2025Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Networkcited by 6
- 2025Ensemble Kalman filter in latent space using a variational autoencoder paircited by 5
- 2025Align-DA: Align Score-based Atmospheric Data Assimilation with Multiple Preferencescited by 5
- 2023Online Calibration of Deep Learning Sub-Models for Hybrid Numerical Modeling Systemscited by 4
- 2023Online machine-learning forecast uncertainty estimation for sequential data assimilationcited by 2
- 2024Ensemble Transport Filter via Optimized Maximum Mean Discrepancycited by 2
- 2024Combined Optimization of Dynamics and Assimilation with End-to-End Learning on Sparse Observationscited by 2
- 2024Development of an offline and online hybrid model for the Integrated Forecasting Systemcited by 1
- 2024Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1cited by 1
- 2025Uncertainty-aware data assimilation through variational inferencecited by 1
- 2025Integrating Score-Based Diffusion Models with Machine Learning-Enhanced Localization for Advanced Data Assimilation in Geological Carbon Storagecited by 1
- 2026ADDA: a Modular Framework for Representing, Simulating and Assimilating Dynamics with End-to-end Differentiabilitycited by 1
- 2021Machine learning for Earth System Science (ESS): A survey, status and future directions for South Asiacited by 0
- 2025Hybrid machine learning data assimilation for marine biogeochemistrycited by 0
- 2026Ensemble Controlled-Flow Filtering for Implicit Data Assimilationcited by 0
- 2026Structured Neural Modeling of Daily Arctic Sea-Ice Concentration Evolution: Physical-Trajectory-Driven Learning and Forecast-Domain Adaptationcited by 0
- 2026ADDA: a Modular Framework for Representing, Simulating and Assimilating Dynamics with End-to-end Differentiability24 Aug 2026
- 2026Structured Neural Modeling of Daily Arctic Sea-Ice Concentration Evolution: Physical-Trajectory-Driven Learning and Forecast-Domain Adaptation22 Aug 2026
- 2026Ensemble Controlled-Flow Filtering for Implicit Data Assimilation14 Jul 2026
- 2025Integrating Score-Based Diffusion Models with Machine Learning-Enhanced Localization for Advanced Data Assimilation in Geological Carbon Storage7 Nov 2025
- 2025Uncertainty-aware data assimilation through variational inference20 Oct 2025
- 2025Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network11 Jul 2025
- 2025Align-DA: Align Score-based Atmospheric Data Assimilation with Multiple Preferences28 May 2025
- 2025Hybrid machine learning data assimilation for marine biogeochemistry7 Apr 2025
- 2025Ensemble Kalman filter in latent space using a variational autoencoder pair18 Feb 2025
- 2024Combined Optimization of Dynamics and Assimilation with End-to-End Learning on Sparse Observations11 Sep 2024
- 2024Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble8 Aug 2024
- 2024State-observation augmented diffusion model for nonlinear assimilation with unknown dynamics31 Jul 2024
28 more
- 2024Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.125 Jul 2024
- 2024Global atmospheric data assimilation with multi-modal masked autoencoders16 Jul 2024
- 2024Ensemble Transport Filter via Optimized Maximum Mean Discrepancy16 Jul 2024
- 2024Deep Generative Data Assimilation in Multimodal Setting10 Apr 2024
- 2024Development of an offline and online hybrid model for the Integrated Forecasting System6 Mar 2024
- 2024Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Response27 Feb 2024
- 2024AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storage9 Feb 2024
- 2023Learning About Structural Errors in Models of Complex Dynamical Systems29 Dec 2023
- 2023Online Calibration of Deep Learning Sub-Models for Hybrid Numerical Modeling Systems17 Nov 2023
- 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementation19 Sep 2023
- 2023Score-based Data Assimilation18 Jun 2023
- 2023Online machine-learning forecast uncertainty estimation for sequential data assimilation12 May 2023
- 2023Digital Twins in Wind Energy: Emerging Technologies and Industry-Informed Future Directions16 Apr 2023
- 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review18 Mar 2023
- 2022Online model error correction with neural networks in the incremental 4D-Var framework25 Oct 2022
- 2022Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems9 Jun 2022
- 2022Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effects10 Mar 2022
- 2021Machine learning for Earth System Science (ESS): A survey, status and future directions for South Asia24 Dec 2021
- 2021Observation Error Covariance Specification in Dynamical Systems for Data assimilation using Recurrent Neural Networks11 Nov 2021
- 2021Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case1 Oct 2021
- 2021Super-resolution data assimilation4 Sep 2021
- 2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations8 Aug 2021
- 2021State, global and local parameter estimation using local ensemble Kalman filters: applications to online machine learning of chaotic dynamics23 Jul 2021
- 2021A comparison of combined data assimilation and machine learning methods for offline and online model error correction23 Jul 2021
- 2021A Framework for Machine Learning of Model Error in Dynamical Systems14 Jul 2021
- 2021Bridging observation, theory and numerical simulation of the ocean using Machine Learning26 Apr 2021
- 2021Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers16 Mar 2021
- 2021Variational Data Assimilation with a Learned Inverse Observation Operator22 Feb 2021
- 2024State-observation augmented diffusion model for nonlinear assimilation with unknown dynamicscited by 7
- 2025Align-DA: Align Score-based Atmospheric Data Assimilation with Multiple Preferencescited by 5
- 2025Integrating Score-Based Diffusion Models with Machine Learning-Enhanced Localization for Advanced Data Assimilation in Geological Carbon Storagecited by 1
- 2025Integrating Score-Based Diffusion Models with Machine Learning-Enhanced Localization for Advanced Data Assimilation in Geological Carbon Storage7 Nov 2025
- 2025Align-DA: Align Score-based Atmospheric Data Assimilation with Multiple Preferences28 May 2025
- 2024State-observation augmented diffusion model for nonlinear assimilation with unknown dynamics31 Jul 2024
- 2023Score-based Data Assimilationcited by 91
- 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 74
- 2021A Framework for Machine Learning of Model Error in Dynamical Systemscited by 37
- 2021Variational Data Assimilation with a Learned Inverse Observation Operatorcited by 28
- 2024Deep Generative Data Assimilation in Multimodal Settingcited by 28
- 2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observationscited by 20
- 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 19
- 2021Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformerscited by 15
- 2021Bridging observation, theory and numerical simulation of the ocean using Machine Learningcited by 15
- 2022Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effectscited by 15
- 2021A comparison of combined data assimilation and machine learning methods for offline and online model error correctioncited by 13
- 2022Online model error correction with neural networks in the incremental 4D-Var frameworkcited by 13
12 most cited of 37 All 37 (JSON)
- 2026ADDA: a Modular Framework for Representing, Simulating and Assimilating Dynamics with End-to-end Differentiability24 Aug 2026
- 2026Structured Neural Modeling of Daily Arctic Sea-Ice Concentration Evolution: Physical-Trajectory-Driven Learning and Forecast-Domain Adaptation22 Aug 2026
- 2026Ensemble Controlled-Flow Filtering for Implicit Data Assimilation14 Jul 2026
- 2025Uncertainty-aware data assimilation through variational inference20 Oct 2025
- 2025Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network11 Jul 2025
- 2025Hybrid machine learning data assimilation for marine biogeochemistry7 Apr 2025
- 2025Ensemble Kalman filter in latent space using a variational autoencoder pair18 Feb 2025
- 2024Combined Optimization of Dynamics and Assimilation with End-to-End Learning on Sparse Observations11 Sep 2024
- 2024Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble8 Aug 2024
- 2024Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.125 Jul 2024
- 2024Global atmospheric data assimilation with multi-modal masked autoencoders16 Jul 2024
- 2024Ensemble Transport Filter via Optimized Maximum Mean Discrepancy16 Jul 2024
12 newest of 37 All 37 (JSON)