mlsanityData to 7 Oct 2026

Stochastic Latent Transformer: Efficient Modelling of Stochastically Forced Zonal Jets

arXiv 2310.16741 ↗First posted 25 Oct 2023cs.LG physics.ao-ph physics.flu-dyncited by 8

Not on the map. Why?

Abstract (arXiv)

We present a novel probabilistic deep learning approach, the 'Stochastic Latent Transformer' (SLT), designed for the efficient reduced-order modelling of stochastic partial differential equations. Stochastically driven flow models are pertinent to a diverse range of natural phenomena, including jets on giant planets, ocean circulation, and the variability of midlatitude weather. However, much of the recent progress in deep learning has predominantly focused on deterministic systems. The SLT comprises a stochastically-forced transformer paired with a translation-equivariant autoencoder, trained towards the Continuous Ranked Probability Score. We showcase its effectiveness by applying it to a well-researched zonal jet system, where the interaction between stochastically forced eddies and the zonal mean flow results in a rich low-frequency variability. The SLT accurately reproduces system dynamics across various integration periods, validated through quantitative diagnostics that include spectral properties and the rate of transitions between distinct states. The SLT achieves a five-order-of-magnitude speedup in emulating the zonally-averaged flow compared to direct numerical simulations. This acceleration facilitates the cost-effective generation of large ensembles, enabling the exploration of statistical questions concerning the probabilities of spontaneous transition events.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 69 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

27 of 69 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  7. 2018On the Spectral Bias of Neural Networkscited by 1,321
  8. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  9. 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 491
  10. 2023ClimaX: A foundation model for weather and climatecited by 325
  11. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 308
  12. 2019Time2Vec: Learning a Vector Representation of Timecited by 208
15 more
  1. 2021Enhancing Computational Fluid Dynamics with Machine Learningcited by 137
  2. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  3. 2020Multiscale Simulations of Complex Systems by Learning their Effective Dynamicscited by 86
  4. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  5. 2023β-Variational autoencoders and transformers for reduced-order modelling of fluid flowscited by 76
  6. 2019Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Modelcited by 45
  7. 2021Data-driven discovery of intrinsic dynamicscited by 45
  8. 2019Deep learning to discover and predict dynamics on an inertial manifoldcited by 42
  9. 2021Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translationcited by 35
  10. 2021Probabilistic Forecasting with Generative Networks via Scoring Rule Minimizationcited by 35
  11. 2022Discovering and forecasting extreme events via active learning in neural operatorscited by 32
  12. 2022Machine learning emulation of a local-scale UK climate modelcited by 20
  13. 2021Convolutional Neural Networks Are Not Invariant to Translation, but They Can Learn to Becited by 19
  14. 2022Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEscited by 11
  15. 2020The Monte Carlo Transformer: a stochastic self-attention model for sequence predictioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2026Demystifying Data-Driven Probabilistic Medium-Range Weather Forecastingcited by 12
  2. 2025A multi-scale loss formulation for learning a probabilistic model with proper score optimisationcited by 6
  3. 2025Causal Climate Emulation with Bayesian Filteringcited by 5
  4. 2026Hierarchy of extreme-event predictability in turbulence revealed by machine learningcited by 2
  5. 2025UP-dROM : Uncertainty-Aware and Parametrised dynamic Reduced-Order Model, application to unsteady flowscited by 0
  6. 2025AttentionDrop: A Novel Regularization Method for Transformer Modelscited by 0
  7. 2025Conditioning on PDE Parameters to Generalise Deep Learning Emulation of Stochastic and Chaotic Dynamicscited by 0
  8. 2026Efficient Adaptation of ROMs for Unsteady Flows Using Data Assimilationcited by 0