mlsanityData to 14 Oct 2026

AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning

arXiv 2308.13280 ↗First posted 25 Aug 2023physics.ao-ph cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

The atmosphere affects humans in a multitude of ways, from loss of life due to adverse weather effects to long-term social and economic impacts on societies. Computer simulations of atmospheric dynamics are, therefore, of great importance for the well-being of our and future generations. Here, we propose AtmoRep, a novel, task-independent stochastic computer model of atmospheric dynamics that can provide skillful results for a wide range of applications. AtmoRep uses large-scale representation learning from artificial intelligence to determine a general description of the highly complex, stochastic dynamics of the atmosphere from the best available estimate of the system's historical trajectory as constrained by observations. This is enabled by a novel self-supervised learning objective and a unique ensemble that samples from the stochastic model with a variability informed by the one in the historical record. The task-independent nature of AtmoRep enables skillful results for a diverse set of applications without specifically training for them and we demonstrate this for nowcasting, temporal interpolation, model correction, and counterfactuals. We also show that AtmoRep can be improved with additional data, for example radar observations, and that it can be extended to tasks such as downscaling. Our work establishes that large-scale neural networks can provide skillful, task-independent models of atmospheric dynamics. With this, they provide a novel means to make the large record of atmospheric observations accessible for applications and for scientific inquiry, complementing existing simulations based on first principles.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 28 of its 90 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

28 of 90 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
16 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021Scaling Vision Transformerscited by 999
  3. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 776
  4. 2020Quantifying Attention Flow in Transformerscited by 775
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  6. 2019Is Attention Interpretable?cited by 389
  7. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  8. 2023ClimaX: A foundation model for weather and climatecited by 337
  9. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 311
  10. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 267
  11. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 241
  12. 2023FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Leadcited by 175
  13. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 152
  14. 2020MetNet: A Neural Weather Model for Precipitation Forecastingcited by 131
  15. 2018Neural networks for post-processing ensemble weather forecastscited by 95
  16. 2021Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction modelscited by 79

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 8; 2026: 14.

40 papers.

Sort:

By area:

  1. 2025FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scalecited by 62
  2. 2024Probabilistic Weather Forecasting with Hierarchical Graph Neural Networkscited by 30
  3. 2024Fast, Scale-Adaptive, and Uncertainty-Aware Downscaling of Earth System Model Fields with Generative Machine Learningcited by 27
  4. 2024Uncertainty quantification for data-driven weather modelscited by 26
  5. 2024Can AI weather models predict out-of-distribution gray swan tropical cyclones?cited by 26
  6. 2023ClimateSet: A Large-Scale Climate Model Dataset for Machine Learningcited by 25
  7. 2023Graph-based Neural Weather Prediction for Limited Area Modelingcited by 22
  8. 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 21
  9. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 16
  10. 2024Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEscited by 15
  11. 2024A Generative Framework for Probabilistic, Spatiotemporally Coherent Downscaling of Climate Simulationcited by 12
  12. 2025Lossy Neural Compression for Geospatial Analytics: A Reviewcited by 10
28 more
  1. 2024Local Off-Grid Weather Forecasting with Multi-Modal Earth Observation Datacited by 9
  2. 2024GPTCast: a weather language model for precipitation nowcastingcited by 8
  3. 2026Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecastingcited by 8
  4. 2024ArchesWeather: An efficient AI weather forecasting model at 1.5deg resolutioncited by 6
  5. 2024STC-ViT: Spatio Temporal Continuous Vision Transformer for Medium-range Global Weather Forecastingcited by 4
  6. 2024Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Datacited by 4
  7. 2024RAIN: Reinforcement Algorithms for Improving Numerical Weather and Climate Modelscited by 3
  8. 2025Finetuning a Weather Foundation Model with Lightweight Decoders for Unseen Physical Processescited by 3
  9. 2026WIND: Weather Inverse Diffusion for Zero-Shot Atmospheric Modelingcited by 3
  10. 2024A Hybrid Deep-Learning Model for El Ni~no Southern Oscillation in the Low-Data Regimecited by 2
  11. 2025From Hallucination to Reliability: Generative Modeling and the Structure of Scientific Inferencecited by 2
  12. 2026PDE foundation models are skillful AI weather emulators for the Martian atmospherecited by 2
  13. 2024Global Vegetation Modeling with Pre-Trained Weather Transformerscited by 1
  14. 2024Identifying Spatio-Temporal Drivers of Extreme Eventscited by 1
  15. 2025Regularization of ML models for Earth systems by using longer model timestepscited by 1
  16. 2025Finetuning AI Foundation Models to Develop Subgrid-Scale Parameterizations: A Case Study on Atmospheric Gravity Wavescited by 1
  17. 2025Field-Space Attention for Structure-Preserving Earth System Transformerscited by 1
  18. 2026Sampling Parallelism for Fast and Efficient Bayesian Learningcited by 1
  19. 2026Towards a Foundation Model for the Martian Atmospherecited by 1
  20. 2026Field-Space Autoencoder for Scalable Climate Emulatorscited by 0
  21. 2026Meteorology-Driven GPT4AP: A Multi-Task Forecasting LLM for Atmospheric Air Pollution in Data-Scarce Settingscited by 0
  22. 2026Accurate, Efficient, and Explainable Deep Learning Approaches for Environmental Science Problemscited by 0
  23. 2026NIVA: A Multimodal Foundation Model for Actionable Earth System Intelligencecited by 0
  24. 2026Precipitation Downscaling Using Foundation Model-Conditioned Diffusioncited by 0
  25. 2026Stochastically Perturbed Weights: Ensembles from Deterministic Machine-Learning Weather Modelscited by 0
  26. 20264D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modelingcited by 0
  27. 2026REACT: Physically and Chemically Consistent Reconstruction of Marine Active Tracerscited by 0
  28. 2026EarthAlign: Learning Language-Aligned Representations of Earth System Datacited by 0