# Forecast Skill Is Not Decision Skill: Evidence from Weather-Dependent Decision Tasks

arXiv: https://arxiv.org/abs/2512.14779 · HTML: https://mlsanity.com/paper/2512.14779/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 13 of 13)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting](https://mlsanity.com/paper/2412.12971/) (2024)
- [Calibration Error for Decision Making](https://mlsanity.com/paper/2404.13503/) (2024)
- [Probabilistic forecasting of German electricity imbalance prices](https://mlsanity.com/paper/2205.11439/) (2022)
- [Smooth Calibration and Decision Making](https://mlsanity.com/paper/2504.15582/) (2025)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Day-Ahead Bidding Strategies for Wind Farm Operators under a One-Price Balancing Scheme](https://mlsanity.com/paper/2505.05153/) (2025)

## Built on by (most cited first, 3 of 3)

- [Probabilistic Transformers for Joint Modeling of Global Weather Dynamics and Decision-Centric Variables](https://mlsanity.com/paper/2601.03753/) (2026)
- [From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol](https://mlsanity.com/paper/2606.24996/) (2026)
- [The Operational Value of Spatial Dependence in Renewable Forecast Scenarios for Single-Period Economic Dispatch: A Controlled Ablation Study](https://mlsanity.com/paper/2609.23988/) (2026)

## Abstract (arXiv)

Standard weather forecast evaluations focus on the forecaster's perspective and on a statistical assessment comparing forecasts and observations. In practice, however, forecasts are used to make decisions, so it seems natural to take the decision-maker's perspective and quantify the value of a forecast by its ability to improve decision-making. Decision calibration provides a novel framework for evaluating probabilistic forecast performance at the decision level rather than the forecast level. We evaluate decision calibration to compare a Machine Learning and a classical numerical weather prediction model on various weather-dependent decision tasks, though the framework is applicable to any set of forecast models. We find that model performance at the forecast level does not reliably translate to performance in downstream decision-making: some performance differences only become apparent at the decision level, and even among seemingly similar decision tasks, model rankings can change. Our results confirm that typical forecast evaluations are insufficient for selecting the optimal forecast model for a specific decision task.