# Scaling transformer neural networks for skillful and reliable medium-range weather forecasting

arXiv: https://arxiv.org/abs/2312.03876 · HTML: https://mlsanity.com/paper/2312.03876/

## Key facts

- Cited by 110 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [MetNet: A Neural Weather Model for Precipitation Forecasting](https://mlsanity.com/paper/2003.12140/) (2020)
- [Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere](https://mlsanity.com/paper/2003.11927/) (2020)
- [Deep Learning for Day Forecasts from Sparse Observations](https://mlsanity.com/paper/2306.06079/) (2023)
- [Combining distribution-based neural networks to predict weather forecast probabilities](https://mlsanity.com/paper/2103.14430/) (2021)

## Built on by (most cited first, 30 of 110)

- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data](https://mlsanity.com/paper/2402.05892/) (2024)
- [Kolmogorov-Arnold Network for Satellite Image Classification in Remote Sensing](https://mlsanity.com/paper/2406.00600/) (2024)
- [ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction](https://mlsanity.com/paper/2402.00712/) (2024)
- [On the Foundations of Earth and Climate Foundation Models](https://mlsanity.com/paper/2405.04285/) (2024)
- [Prithvi WxC: Foundation Model for Weather and Climate](https://mlsanity.com/paper/2409.13598/) (2024)
- [Probabilistic Weather Forecasting with Hierarchical Graph Neural Networks](https://mlsanity.com/paper/2406.04759/) (2024)
- [Continuous Ensemble Weather Forecasting with Diffusion models](https://mlsanity.com/paper/2410.05431/) (2024)
- [Foundation Models for the Electric Power Grid](https://mlsanity.com/paper/2407.09434/) (2024)
- [Uncertainty quantification for data-driven weather models](https://mlsanity.com/paper/2403.13458/) (2024)
- [Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling](https://mlsanity.com/paper/2405.13796/) (2024)
- [WeatherQA: Can Multimodal Language Models Reason about Severe Weather?](https://mlsanity.com/paper/2406.11217/) (2024)
- [Probabilistic Emulation of a Global Climate Model with Spherical DYffusion](https://mlsanity.com/paper/2406.14798/) (2024)
- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability](https://mlsanity.com/paper/2404.14712/) (2024)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction](https://mlsanity.com/paper/2404.19630/) (2024)
- [Demystifying Data-Driven Probabilistic Medium-Range Weather Forecasting](https://mlsanity.com/paper/2601.18111/) (2026)
- [Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events](https://mlsanity.com/paper/2404.17652/) (2024)
- [Elucidated Rolling Diffusion Models for Probabilistic Forecasting of Complex Dynamics](https://mlsanity.com/paper/2506.20024/) (2025)
- [Swift: An Autoregressive Consistency Model for Efficient Weather Forecasting](https://mlsanity.com/paper/2509.25631/) (2025)
- [NeuralOM: Neural Ocean Model for Subseasonal-to-Seasonal Simulation](https://mlsanity.com/paper/2505.21020/) (2025)
- [OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales](https://mlsanity.com/paper/2510.18707/) (2025)
- [U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster](https://mlsanity.com/paper/2604.09041/) (2026)
- [Robustness of AI-based weather forecasts in a changing climate](https://mlsanity.com/paper/2409.18529/) (2024)
- [Weather Prediction with Diffusion Guided by Realistic Forecast Processes](https://mlsanity.com/paper/2402.06666/) (2024)
- [Binned Spectral Power Loss for Improved Prediction of Chaotic Systems](https://mlsanity.com/paper/2502.00472/) (2025)
- [Zephyrus: An Agentic Framework for Weather Science](https://mlsanity.com/paper/2510.04017/) (2025)
- [Scaling Laws of Global Weather Models](https://mlsanity.com/paper/2602.22962/) (2026)
- [EWMoE: An effective model for global weather forecasting with mixture-of-experts](https://mlsanity.com/paper/2405.06004/) (2024)

## Abstract (arXiv)

Weather forecasting is a fundamental problem for anticipating and mitigating the impacts of climate change. Recently, data-driven approaches for weather forecasting based on deep learning have shown great promise, achieving accuracies that are competitive with operational systems. However, those methods often employ complex, customized architectures without sufficient ablation analysis, making it difficult to understand what truly contributes to their success. Here we introduce Stormer, a simple transformer model that achieves state-of-the-art performance on weather forecasting with minimal changes to the standard transformer backbone. We identify the key components of Stormer through careful empirical analyses, including weather-specific embedding, randomized dynamics forecast, and pressure-weighted loss. At the core of Stormer is a randomized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals. During inference, this allows us to produce multiple forecasts for a target lead time and combine them to obtain better forecast accuracy. On WeatherBench 2, Stormer performs competitively at short to medium-range forecasts and outperforms current methods beyond 7 days, while requiring orders-of-magnitude less training data and compute. Additionally, we demonstrate Stormer's favorable scaling properties, showing consistent improvements in forecast accuracy with increases in model size and training tokens. Code and checkpoints are available at https://github.com/tung-nd/stormer.