# AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning

arXiv: https://arxiv.org/abs/2308.13280 · HTML: https://mlsanity.com/paper/2308.13280/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 90 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Understanding Diffusion Models: A Unified Perspective](https://mlsanity.com/paper/2208.11970/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [MetNet: A Neural Weather Model for Precipitation Forecasting](https://mlsanity.com/paper/2003.12140/) (2020)
- [Neural networks for post-processing ensemble weather forecasts](https://mlsanity.com/paper/1805.09091/) (2018)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)

## Built on by (most cited first, 30 of 40)

- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [Probabilistic Weather Forecasting with Hierarchical Graph Neural Networks](https://mlsanity.com/paper/2406.04759/) (2024)
- [Fast, Scale-Adaptive, and Uncertainty-Aware Downscaling of Earth System Model Fields with Generative Machine Learning](https://mlsanity.com/paper/2403.02774/) (2024)
- [Uncertainty quantification for data-driven weather models](https://mlsanity.com/paper/2403.13458/) (2024)
- [Can AI weather models predict out-of-distribution gray swan tropical cyclones?](https://mlsanity.com/paper/2410.14932/) (2024)
- [ClimateSet: A Large-Scale Climate Model Dataset for Machine Learning](https://mlsanity.com/paper/2311.03721/) (2023)
- [Graph-based Neural Weather Prediction for Limited Area Modeling](https://mlsanity.com/paper/2309.17370/) (2023)
- [AI Foundation Models for Weather and Climate: Applications, Design, and Implementation](https://mlsanity.com/paper/2309.10808/) (2023)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEs](https://mlsanity.com/paper/2403.10642/) (2024)
- [A Generative Framework for Probabilistic, Spatiotemporally Coherent Downscaling of Climate Simulation](https://mlsanity.com/paper/2412.15361/) (2024)
- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [Local Off-Grid Weather Forecasting with Multi-Modal Earth Observation Data](https://mlsanity.com/paper/2410.12938/) (2024)
- [GPTCast: a weather language model for precipitation nowcasting](https://mlsanity.com/paper/2407.02089/) (2024)
- [Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecasting](https://mlsanity.com/paper/2605.00850/) (2026)
- [ArchesWeather: An efficient AI weather forecasting model at 1.5deg resolution](https://mlsanity.com/paper/2405.14527/) (2024)
- [STC-ViT: Spatio Temporal Continuous Vision Transformer for Medium-range Global Weather Forecasting](https://mlsanity.com/paper/2402.17966/) (2024)
- [Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Data](https://mlsanity.com/paper/2410.06731/) (2024)
- [RAIN: Reinforcement Algorithms for Improving Numerical Weather and Climate Models](https://mlsanity.com/paper/2408.16118/) (2024)
- [Finetuning a Weather Foundation Model with Lightweight Decoders for Unseen Physical Processes](https://mlsanity.com/paper/2506.19088/) (2025)
- [WIND: Weather Inverse Diffusion for Zero-Shot Atmospheric Modeling](https://mlsanity.com/paper/2602.03924/) (2026)
- [A Hybrid Deep-Learning Model for El Ni~no Southern Oscillation in the Low-Data Regime](https://mlsanity.com/paper/2412.03743/) (2024)
- [From Hallucination to Reliability: Generative Modeling and the Structure of Scientific Inference](https://mlsanity.com/paper/2504.08526/) (2025)
- [PDE foundation models are skillful AI weather emulators for the Martian atmosphere](https://mlsanity.com/paper/2602.15004/) (2026)
- [Global Vegetation Modeling with Pre-Trained Weather Transformers](https://mlsanity.com/paper/2403.18438/) (2024)
- [Identifying Spatio-Temporal Drivers of Extreme Events](https://mlsanity.com/paper/2410.24075/) (2024)
- [Regularization of ML models for Earth systems by using longer model timesteps](https://mlsanity.com/paper/2503.18023/) (2025)
- [Finetuning AI Foundation Models to Develop Subgrid-Scale Parameterizations: A Case Study on Atmospheric Gravity Waves](https://mlsanity.com/paper/2509.03816/) (2025)
- [Field-Space Attention for Structure-Preserving Earth System Transformers](https://mlsanity.com/paper/2512.20350/) (2025)
- [Sampling Parallelism for Fast and Efficient Bayesian Learning](https://mlsanity.com/paper/2604.04736/) (2026)

## Abstract (arXiv)

The atmosphere affects humans in a multitude of ways, from loss of life due to adverse weather effects to long-term social and economic impacts on societies. Computer simulations of atmospheric dynamics are, therefore, of great importance for the well-being of our and future generations. Here, we propose AtmoRep, a novel, task-independent stochastic computer model of atmospheric dynamics that can provide skillful results for a wide range of applications. AtmoRep uses large-scale representation learning from artificial intelligence to determine a general description of the highly complex, stochastic dynamics of the atmosphere from the best available estimate of the system's historical trajectory as constrained by observations. This is enabled by a novel self-supervised learning objective and a unique ensemble that samples from the stochastic model with a variability informed by the one in the historical record. The task-independent nature of AtmoRep enables skillful results for a diverse set of applications without specifically training for them and we demonstrate this for nowcasting, temporal interpolation, model correction, and counterfactuals. We also show that AtmoRep can be improved with additional data, for example radar observations, and that it can be extended to tasks such as downscaling. Our work establishes that large-scale neural networks can provide skillful, task-independent models of atmospheric dynamics. With this, they provide a novel means to make the large record of atmospheric observations accessible for applications and for scientific inquiry, complementing existing simulations based on first principles.