# WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models

arXiv: https://arxiv.org/abs/2205.00865 · HTML: https://mlsanity.com/paper/2205.00865/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 14 of 14)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [MetNet: A Neural Weather Model for Precipitation Forecasting](https://mlsanity.com/paper/2003.12140/) (2020)
- [Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere](https://mlsanity.com/paper/2003.11927/) (2020)
- [Neural networks for post-processing ensemble weather forecasts](https://mlsanity.com/paper/1805.09091/) (2018)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)
- [A Generative Deep Learning Approach to Stochastic Downscaling of Precipitation Forecasts](https://mlsanity.com/paper/2204.02028/) (2022)
- [Increasing the accuracy and resolution of precipitation forecasts using deep generative models](https://mlsanity.com/paper/2203.12297/) (2022)
- [Skillful Twelve Hour Precipitation Forecasts using Large Context Neural Networks](https://mlsanity.com/paper/2111.07470/) (2021)
- [Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)](https://mlsanity.com/paper/1912.01956/) (2019)
- [Ensemble methods for neural network-based weather forecasts](https://mlsanity.com/paper/2002.05398/) (2020)

## Built on by (most cited first, 17 of 17)

- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting](https://mlsanity.com/paper/2306.01984/) (2023)
- [ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling](https://mlsanity.com/paper/2307.01909/) (2023)
- [SwinVRNN: A Data-Driven Ensemble Forecasting Model via Learned Distribution Perturbation](https://mlsanity.com/paper/2205.13158/) (2022)
- [SwinRDM: Integrate SwinRNN with Diffusion Model towards High-Resolution and High-Quality Weather Forecasting](https://mlsanity.com/paper/2306.03110/) (2023)
- [Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciar'an](https://mlsanity.com/paper/2312.02658/) (2023)
- [U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster](https://mlsanity.com/paper/2604.09041/) (2026)
- [PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines](https://mlsanity.com/paper/2407.08418/) (2024)
- [ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method](https://mlsanity.com/paper/2504.07394/) (2025)
- [Inductive biases in deep learning models for weather prediction](https://mlsanity.com/paper/2304.04664/) (2023)
- [On the Predictive Skill of Artificial Intelligence-based Weather Models for Extreme Events using Uncertainty Quantification](https://mlsanity.com/paper/2511.17176/) (2025)
- [Workload Forecasting of a Logistic Node Using Bayesian Neural Networks](https://mlsanity.com/paper/2211.04976/) (2022)
- [RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges](https://mlsanity.com/paper/2605.24945/) (2026)
- [Reliability of Probabilistic Emulation of Physical Systems](https://mlsanity.com/paper/2606.12997/) (2026)
- [Stochastically Perturbed Weights: Ensembles from Deterministic Machine-Learning Weather Models](https://mlsanity.com/paper/2609.08412/) (2026)
- [Generative Atmospheric Super-Resolution from Heterogeneous In Situ Observations through Composable Interfaces](https://mlsanity.com/paper/2609.29027/) (2026)

## Abstract (arXiv)

WeatherBench is a benchmark dataset for medium-range weather forecasting of geopotential, temperature and precipitation, consisting of preprocessed data, predefined evaluation metrics and a number of baseline models. WeatherBench Probability extends this to probabilistic forecasting by adding a set of established probabilistic verification metrics (continuous ranked probability score, spread-skill ratio and rank histograms) and a state-of-the-art operational baseline using the ECWMF IFS ensemble forecast. In addition, we test three different probabilistic machine learning methods -- Monte Carlo dropout, parametric prediction and categorical prediction, in which the probability distribution is discretized. We find that plain Monte Carlo dropout severely underestimates uncertainty. The parametric and categorical models both produce fairly reliable forecasts of similar quality. The parametric models have fewer degrees of freedom while the categorical model is more flexible when it comes to predicting non-Gaussian distributions. None of the models are able to match the skill of the operational IFS model. We hope that this benchmark will enable other researchers to evaluate their probabilistic approaches.