# Probabilistic measures afford fair comparisons of AIWP and NWP model output

arXiv: https://arxiv.org/abs/2506.03744 · HTML: https://mlsanity.com/paper/2506.03744/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 11 of 11)

- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Isotonic Distributional Regression](https://mlsanity.com/paper/1909.03725/) (2019)
- [Uncertainty quantification for data-driven weather models](https://mlsanity.com/paper/2403.13458/) (2024)
- [A Practical Probabilistic Benchmark for AI Weather Models](https://mlsanity.com/paper/2401.15305/) (2024)
- [Forecaster's Dilemma: Extreme Events and Forecast Evaluation](https://mlsanity.com/paper/1512.09244/) (2015)
- [Easy Uncertainty Quantification (EasyUQ): Generating Predictive Distributions from Single-valued Model Output](https://mlsanity.com/paper/2212.08376/) (2022)
- [Monotone Least Squares and Isotonic Quantiles](https://mlsanity.com/paper/1901.02398/) (2019)
- [Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distributional regression](https://mlsanity.com/paper/2404.02270/) (2024)
- [Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)](https://mlsanity.com/paper/1912.01956/) (2019)

## Built on by (most cited first, 3 of 3)

- [Statistical versus machine learning-based spatial interpolation of post-processed ensemble weather forecasts](https://mlsanity.com/paper/2609.07512/) (2026)
- [Suitable Measures for the Potential Operational Utility of AI NWP Rainfall Forecasts Over Africa](https://mlsanity.com/paper/2609.31775/) (2026)
- [A Free Knob: Decoupling Calibration and Predictive Skill in Threshold-Based Evaluation](https://mlsanity.com/paper/2609.33457/) (2026)

## Abstract (arXiv)

We introduce a new measure for fair and meaningful comparisons of single-valued output from artificial intelligence based weather prediction (AIWP) and numerical weather prediction (NWP) models, called potential continuous ranked probability score (PC). In a nutshell, we subject the deterministic backbone of physics-based and data-driven models post hoc to the same statistical postprocessing technique, namely, isotonic distributional regression (IDR). Then we find PC as the mean continuous ranked probability score (CRPS) of the postprocessed probabilistic forecasts. The nonnegative PC measure quantifies potential predictive performance and is invariant under strictly increasing transformations of the model output. PC attains its most desirable value of zero if, and only if, the weather outcome Y is a fixed, non-decreasing function of the model output X. The PC measure is recorded in the unit of the outcome, has an upper bound of one half times the mean absolute difference between outcomes, and serves as a proxy for the mean CRPS of real-time, operational probabilistic products. When applied to WeatherBench 2 data, our approach demonstrates that the data-driven GraphCast model outperforms the leading, physics-based European Centre for Medium Range Weather Forecasts (ECMWF) high-resolution (HRES) model. Furthermore, the PC measure for the HRES model aligns exceptionally well with the mean CRPS of the operational ECMWF ensemble. Across application domains, our approach affords comparisons of single-valued forecasts in settings where the pre-specification of a loss function -- which is the usual, and principally superior, procedure in forecast contests, administrative, and benchmarks settings -- places competitors on unequal footings.