# Forecast Evaluation of Quantiles, Prediction Intervals, and other Set-Valued Functionals

arXiv: https://arxiv.org/abs/1910.07912 · HTML: https://mlsanity.com/paper/1910.07912/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice](https://mlsanity.com/paper/2202.12780/) (2022)
- [From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions](https://mlsanity.com/paper/2301.11797/) (2023)
- [Ensemble learning for uncertainty estimation with application to the correction of satellite precipitation products](https://mlsanity.com/paper/2403.10567/) (2024)
- [Property Elicitation on Imprecise Probabilities](https://mlsanity.com/paper/2507.05857/) (2025)
- [Hierarchies of Calibration: Classification meets Regression](https://mlsanity.com/paper/2606.03245/) (2026)
- [Combinations of distributional regression algorithms with application in uncertainty estimation of corrected satellite precipitation products](https://mlsanity.com/paper/2407.01623/) (2024)
- [Variable transformations in consistent loss functions](https://mlsanity.com/paper/2502.16542/) (2025)

## Abstract (arXiv)

We introduce a theoretical framework of elicitability and identifiability of set-valued functionals, such as quantiles, prediction intervals, and systemic risk measures. A functional is elicitable if it is the unique minimiser of an expected scoring function, and identifiable if it is the unique zero of an expected identification function; both notions are essential for forecast ranking and validation, and M- and Z-estimation. Our framework distinguishes between exhaustive forecasts, being set-valued and aiming at correctly specifying the entire functional, and selective forecasts, content with solely specifying a single point in the correct functional. We establish a mutual exclusivity result: A set-valued functional can be either selectively elicitable or exhaustively elicitable or not elicitable at all. Notably, since quantiles are well known to be selectively elicitable, they fail to be exhaustively elicitable. We further show that the class of prediction intervals and Vorob'ev quantiles turn out to be exhaustively elicitable and selectively identifiable. In particular, we provide a mixture representation of elementary exhaustive scores, leading the way to Murphy diagrams. We give possibility and impossibility results for the shortest prediction interval and prediction intervals specified by an endpoint or a midpoint. We end with a comprehensive literature review on common practice in forecast evaluation of set-valued functionals.