# Exploring the design space of deep-learning-based weather forecasting systems

arXiv: https://arxiv.org/abs/2410.07472 · HTML: https://mlsanity.com/paper/2410.07472/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 64 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)

## Built on by (most cited first, 8 of 8)

- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [Diffusion-LAM: Probabilistic Limited Area Weather Forecasting with Diffusion](https://mlsanity.com/paper/2502.07532/) (2025)
- [AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions](https://mlsanity.com/paper/2509.13523/) (2025)
- [HRRRCast: a data-driven emulator for regional weather forecasting at convection allowing scales](https://mlsanity.com/paper/2507.05658/) (2025)
- [Community Research Earth Digital Intelligence Twin (CREDIT)](https://mlsanity.com/paper/2411.07814/) (2024)
- [SwAIther-Precip: Lead-Time-Aware Bias Correction Enables Kilometer-Scale Downscaling of Global AI Precipitation Forecasts over Switzerland](https://mlsanity.com/paper/2605.16163/) (2026)
- [Samudra 2: Scaling Ocean Emulators across Resolutions](https://mlsanity.com/paper/2606.02610/) (2026)
- [Artificial intelligence pathways from weather to climate](https://mlsanity.com/paper/2610.09770/) (2026)

## Abstract (arXiv)

Despite tremendous progress in developing deep-learning-based weather forecasting systems, their design space, including the impact of different design choices, is yet to be well understood. This paper aims to fill this knowledge gap by systematically analyzing these choices including architecture, problem formulation, pretraining scheme, use of image-based pretrained models, loss functions, noise injection, multi-step inputs, additional static masks, multi-step finetuning (including larger stride models), as well as training on a larger dataset. We study fixed-grid architectures such as UNet, fully convolutional architectures, and transformer-based models, along with grid-invariant architectures, including graph-based and operator-based models. Our results show that fixed-grid architectures outperform grid-invariant architectures, indicating a need for further architectural developments in grid-invariant models such as neural operators. We therefore propose a hybrid system that combines the strong performance of fixed-grid models with the flexibility of grid-invariant architectures. We further show that multi-step fine-tuning is essential for most deep-learning models to work well in practice, which has been a common practice in the past. Pretraining objectives degrade performance in comparison to supervised training, while image-based pretrained models provide useful inductive biases in some cases in comparison to training the model from scratch. Interestingly, we see a strong positive effect of using a larger dataset when training a smaller model as compared to training on a smaller dataset for longer. Larger models, on the other hand, primarily benefit from just an increase in the computational budget. We believe that these results will aid in the design of better weather forecasting systems in the future.