# Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction

arXiv: https://arxiv.org/abs/2404.19630 · HTML: https://mlsanity.com/paper/2404.19630/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast](https://mlsanity.com/paper/2211.02556/) (2022)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [Scaling transformer neural networks for skillful and reliable medium-range weather forecasting](https://mlsanity.com/paper/2312.03876/) (2023)
- [Advancing Parsimonious Deep Learning Weather Prediction using the HEALPix Mesh](https://mlsanity.com/paper/2311.06253/) (2023)
- [A Practical Probabilistic Benchmark for AI Weather Models](https://mlsanity.com/paper/2401.15305/) (2024)

## Built on by (most cited first, 15 of 15)

- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [Swift: An Autoregressive Consistency Model for Efficient Weather Forecasting](https://mlsanity.com/paper/2509.25631/) (2025)
- [AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions](https://mlsanity.com/paper/2509.13523/) (2025)
- [PEAR: Equal Area Weather Forecasting on the Sphere](https://mlsanity.com/paper/2505.17720/) (2025)
- [Community Research Earth Digital Intelligence Twin (CREDIT)](https://mlsanity.com/paper/2411.07814/) (2024)
- [Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts](https://mlsanity.com/paper/2605.30184/) (2026)
- [MoWE : A Mixture of Weather Experts](https://mlsanity.com/paper/2509.09052/) (2025)
- [Examining Fast Radiatively Driven Responses Using Machine-Learning Weather Emulators](https://mlsanity.com/paper/2602.16090/) (2026)
- [On Neural Scaling Laws for Weather Emulation through Continual Training](https://mlsanity.com/paper/2603.25687/) (2026)
- [Probabilistic Transformers for Joint Modeling of Global Weather Dynamics and Decision-Centric Variables](https://mlsanity.com/paper/2601.03753/) (2026)
- [Comprehensive Performance Modeling and System Design Insights for Foundation Models](https://mlsanity.com/paper/2410.00273/) (2024)
- [Investigating the use of terrain-following coordinates in AI-driven precipitation forecasts](https://mlsanity.com/paper/2503.00332/) (2025)
- [An adaptive and evolvable deep reinforcement learning framework for weather prediction](https://mlsanity.com/paper/2608.09948/) (2026)
- [Aries: A Proprietary Medium-Range Weather Prediction Model for the Energy Industry](https://mlsanity.com/paper/2609.13292/) (2026)
- [ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences](https://mlsanity.com/paper/2609.31918/) (2026)

## Abstract (arXiv)

The rapid rise of deep learning (DL) in numerical weather prediction (NWP) has led to a proliferation of models which forecast atmospheric variables with comparable or superior skill than traditional physics-based NWP. However, among these leading DL models, there is a wide variance in both the training settings and architecture used. Further, the lack of thorough ablation studies makes it hard to discern which components are most critical to success. In this work, we show that it is possible to attain high forecast skill even with relatively off-the-shelf architectures, simple training procedures, and moderate compute budgets. Specifically, we train a minimally modified SwinV2 transformer on ERA5 data, and find that it attains superior forecast skill when compared against IFS. We present some ablations on key aspects of the training pipeline, exploring different loss functions, model sizes and depths, and multi-step fine-tuning to investigate their effect. We also examine the model performance with metrics beyond the typical ACC and RMSE, and investigate how the performance scales with model size.