# Enforcing tail calibration when training probabilistic forecast models

arXiv: https://arxiv.org/abs/2506.13687 · HTML: https://mlsanity.com/paper/2506.13687/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Theoretical Foundations of Conformal Prediction](https://mlsanity.com/paper/2411.11824/) (2024)
- [Neural networks for post-processing ensemble weather forecasts](https://mlsanity.com/paper/1805.09091/) (2018)
- [Probabilistic Forecasting with Generative Networks via Scoring Rule Minimization](https://mlsanity.com/paper/2112.08217/) (2021)
- [Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination](https://mlsanity.com/paper/2108.03210/) (2021)
- [Isotonic Distributional Regression](https://mlsanity.com/paper/1909.03725/) (2019)
- [Generative machine learning methods for multivariate ensemble post-processing](https://mlsanity.com/paper/2211.01345/) (2022)
- [Forecaster's Dilemma: Extreme Events and Forecast Evaluation](https://mlsanity.com/paper/1512.09244/) (2015)
- [Proper scoring rules for estimation and forecast evaluation](https://mlsanity.com/paper/2504.01781/) (2025)
- [Evaluating forecasts for high-impact events using transformed kernel scores](https://mlsanity.com/paper/2202.12732/) (2022)
- [In-sample calibration yields conformal calibration guarantees](https://mlsanity.com/paper/2503.03841/) (2025)
- [Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions](https://mlsanity.com/paper/1905.04022/) (2019)
- [Tail calibration of probabilistic forecasts](https://mlsanity.com/paper/2407.03167/) (2024)
- [Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules](https://mlsanity.com/paper/2407.15900/) (2024)

## Built on by (most cited first, 5 of 5)

- [EnScale: Temporally-consistent multivariate generative downscaling via proper scoring rules](https://mlsanity.com/paper/2509.26258/) (2025)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [Calibrated Multivariate Distributional Regression with Pre-Rank Regularization](https://mlsanity.com/paper/2601.22895/) (2026)
- [Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization](https://mlsanity.com/paper/2510.21273/) (2025)
- [Training ML Models with Predictable Failures](https://mlsanity.com/paper/2605.15134/) (2026)

## Abstract (arXiv)

Probabilistic forecasts are typically obtained using state-of-the-art statistical and machine learning models, with model parameters estimated by optimizing a proper scoring rule over a set of training data. If the model class is not correctly specified, then the learned model will not necessarily issue forecasts that are calibrated. Calibrated forecasts allow users to appropriately balance risks in decision making, and it is particularly important that forecast models issue calibrated predictions for extreme events, since such outcomes often generate large socio-economic impacts. In this work, we study how the loss function used to train probabilistic forecast models can be adapted to improve the reliability of forecasts made for extreme events. We investigate loss functions based on weighted scoring rules, and additionally propose regularizing loss functions using a measure of tail miscalibration. We apply these approaches to a hierarchy of increasingly flexible forecast models for UK wind speeds, including simple parametric models, distributional regression networks, and conditional generative models. We demonstrate that state-of-the-art models do not issue calibrated forecasts for extreme wind speeds, and that the calibration of forecasts for extreme events can be improved by suitable adaptations to the loss function during model training. This introduces a trade-off between calibrated forecasts for extreme events and calibrated forecasts for more common outcomes.