# Evaluating forecasts for high-impact events using transformed kernel scores

arXiv: https://arxiv.org/abs/2202.12732 · HTML: https://mlsanity.com/paper/2202.12732/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2022.

## Built on by (most cited first, 11 of 11)

- [Generative machine learning methods for multivariate ensemble post-processing](https://mlsanity.com/paper/2211.01345/) (2022)
- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [Proper scoring rules for estimation and forecast evaluation](https://mlsanity.com/paper/2504.01781/) (2025)
- [Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events](https://mlsanity.com/paper/2404.17652/) (2024)
- [Enforcing tail calibration when training probabilistic forecast models](https://mlsanity.com/paper/2506.13687/) (2025)
- [Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules](https://mlsanity.com/paper/2407.15900/) (2024)
- [CRPS-Based Targeted Sequential Design with Application in Chemical Space](https://mlsanity.com/paper/2503.11250/) (2025)
- [Uncertainty Quantification for Regression: A Unified Framework based on kernel scores](https://mlsanity.com/paper/2510.25599/) (2025)
- [Aligning the Evaluation of Probabilistic Predictions with Downstream Value](https://mlsanity.com/paper/2508.18251/) (2025)
- [LLM as Forecasting Planner: Training-Free Text Conditioning for Time-Series Foundation Models](https://mlsanity.com/paper/2607.24892/) (2026)
- [Efficient pooling of predictions via kernel embeddings](https://mlsanity.com/paper/2411.16246/) (2024)

## Abstract (arXiv)

It is informative to evaluate a forecaster's ability to predict outcomes that have a large impact on the forecast user. Although weighted scoring rules have become a well-established tool to achieve this, such scores have been studied almost exclusively in the univariate case, with interest typically placed on extreme events. However, a large impact may also result from events not considered to be extreme from a statistical perspective: the interaction of several moderate events could also generate a high impact. Compound weather events provide a good example of this. To assess forecasts made for high-impact events, this work extends existing results on weighted scoring rules by introducing weighted multivariate scores. To do so, we utilise kernel scores. We demonstrate that the threshold-weighted continuous ranked probability score (twCRPS), arguably the most well-known weighted scoring rule, is a kernel score. This result leads to a convenient representation of the twCRPS when the forecast is an ensemble, and also permits a generalisation that can be employed with alternative kernels, allowing us to introduce, for example, a threshold-weighted energy score and threshold-weighted variogram score. To illustrate the additional information that these weighted multivariate scoring rules provide, results are presented for a case study in which the weighted scores are used to evaluate daily precipitation accumulation forecasts, with particular interest on events that could lead to flooding.