# A Practical Probabilistic Benchmark for AI Weather Models

arXiv: https://arxiv.org/abs/2401.15305 · HTML: https://mlsanity.com/paper/2401.15305/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 27 Jan 2024.

## Builds on (most cited first, 10 of 10)

- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere](https://mlsanity.com/paper/2003.11927/) (2020)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)

## Built on by (most cited first, 25 of 25)

- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function](https://mlsanity.com/paper/2501.19374/) (2025)
- [Uncertainty quantification for data-driven weather models](https://mlsanity.com/paper/2403.13458/) (2024)
- [Coupled Ocean-Atmosphere Dynamics in a Machine Learning Earth System Model](https://mlsanity.com/paper/2406.08632/) (2024)
- [Probabilistic Emulation of a Global Climate Model with Spherical DYffusion](https://mlsanity.com/paper/2406.14798/) (2024)
- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction](https://mlsanity.com/paper/2404.19630/) (2024)
- [FuXi-ENS: A machine learning model for medium-range ensemble weather forecasting](https://mlsanity.com/paper/2405.05925/) (2024)
- [U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster](https://mlsanity.com/paper/2604.09041/) (2026)
- [Atmospheric Predictability Beyond 30 Days with Machine Learning](https://mlsanity.com/paper/2504.20238/) (2025)
- [Community Research Earth Digital Intelligence Twin (CREDIT)](https://mlsanity.com/paper/2411.07814/) (2024)
- [MAUSAM: An Observations-focused assessment of Global AI Weather Prediction Models During the South Asian Monsoon](https://mlsanity.com/paper/2509.01879/) (2025)
- [Probabilistic measures afford fair comparisons of AIWP and NWP model output](https://mlsanity.com/paper/2506.03744/) (2025)
- [MoWE : A Mixture of Weather Experts](https://mlsanity.com/paper/2509.09052/) (2025)
- [Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models](https://mlsanity.com/paper/2512.22814/) (2025)
- [Examining Fast Radiatively Driven Responses Using Machine-Learning Weather Emulators](https://mlsanity.com/paper/2602.16090/) (2026)
- [On the Predictive Skill of Artificial Intelligence-based Weather Models for Extreme Events using Uncertainty Quantification](https://mlsanity.com/paper/2511.17176/) (2025)
- [Potential Paradigm Shift in Hazard Risk Management: AI-Based Weather Forecast for Tropical Cyclone Hazards](https://mlsanity.com/paper/2404.18440/) (2024)
- [Comprehensive Performance Modeling and System Design Insights for Foundation Models](https://mlsanity.com/paper/2410.00273/) (2024)
- [Hierarchical Testing of a Hybrid Machine Learning-Physics Global Atmosphere Model](https://mlsanity.com/paper/2602.11313/) (2026)
- [Spatio-temporal probabilistic forecast using MMAF-guided learning](https://mlsanity.com/paper/2603.15055/) (2026)
- [On a joint simultaneous learning of relevant feature subsets and subspaces in regression-like problems](https://mlsanity.com/paper/2607.28080/) (2026)
- [Improving global precipitation forecasts with an AI weather model trained on satellite observations](https://mlsanity.com/paper/2609.03210/) (2026)
- [Lightweight Probabilistic Downscaling from a Deterministic Base Model](https://mlsanity.com/paper/2609.29383/) (2026)
- [Suitable Measures for the Potential Operational Utility of AI NWP Rainfall Forecasts Over Africa](https://mlsanity.com/paper/2609.31775/) (2026)

## Abstract (arXiv)

Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.