# Forecaster's Dilemma: Extreme Events and Forecast Evaluation

arXiv: https://arxiv.org/abs/1512.09244 · HTML: https://mlsanity.com/paper/1512.09244/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2015.

## Built on by (most cited first, 20 of 20)

- [Generative machine learning methods for multivariate ensemble post-processing](https://mlsanity.com/paper/2211.01345/) (2022)
- [Uncertainty quantification for data-driven weather models](https://mlsanity.com/paper/2403.13458/) (2024)
- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events](https://mlsanity.com/paper/2404.17652/) (2024)
- [Numerical models outperform AI weather forecasts of record-breaking extremes](https://mlsanity.com/paper/2508.15724/) (2025)
- [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice](https://mlsanity.com/paper/2202.12780/) (2022)
- [Global Extreme Heat Forecasting Using Neural Weather Models](https://mlsanity.com/paper/2205.10972/) (2022)
- [A review of machine learning concepts and methods for addressing challenges in probabilistic hydrological post-processing and forecasting](https://mlsanity.com/paper/2206.08998/) (2022)
- [Huge Ensembles Part II: Properties of a Huge Ensemble of Hindcasts Generated with Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.01581/) (2024)
- [Can Machine Learning Catch the COVID-19 Recession?](https://mlsanity.com/paper/2103.01201/) (2021)
- [Enforcing tail calibration when training probabilistic forecast models](https://mlsanity.com/paper/2506.13687/) (2025)
- [Probabilistic measures afford fair comparisons of AIWP and NWP model output](https://mlsanity.com/paper/2506.03744/) (2025)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather](https://mlsanity.com/paper/2605.01126/) (2026)
- [Distributional Regression U-Nets for the Postprocessing of Precipitation Ensemble Forecasts](https://mlsanity.com/paper/2407.02125/) (2024)
- [Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules](https://mlsanity.com/paper/2407.15900/) (2024)
- [Quantifying the Risk-Return Tradeoff in Forecasting](https://mlsanity.com/paper/2605.09712/) (2026)
- [Forecasting threshold exceedance of atmospheric variables at a specific location](https://mlsanity.com/paper/2605.31079/) (2026)
- [Decision-Aware Training for Sample-Based Generative Models](https://mlsanity.com/paper/2607.01171/) (2026)
- [Asymmetric Peak-Aware Loss for Peak-Critical Time Series Forecasting](https://mlsanity.com/paper/2607.14871/) (2026)

## Abstract (arXiv)

In public discussions of the quality of forecasts, attention typically focuses on the predictive performance in cases of extreme events. However, the restriction of conventional forecast evaluation methods to subsets of extreme observations has unexpected and undesired effects, and is bound to discredit skillful forecasts when the signal-to-noise ratio in the data generating process is low. Conditioning on outcomes is incompatible with the theoretical assumptions of established forecast evaluation methods, thereby confronting forecasters with what we refer to as the forecaster's dilemma. For probabilistic forecasts, proper weighted scoring rules have been proposed as decision theoretically justifiable alternatives for forecast evaluation with an emphasis on extreme events. Using theoretical arguments, simulation experiments, and a real data study on probabilistic forecasts of U.S. inflation and gross domestic product growth, we illustrate and discuss the forecaster's dilemma along with potential remedies.