# From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions

arXiv: https://arxiv.org/abs/2301.11797 · HTML: https://mlsanity.com/paper/2301.11797/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 27 Jan 2023.

## Builds on (most cited first, 8 of 8)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [On the Calibration of Multiclass Classification with Rejection](https://mlsanity.com/paper/1901.10655/) (2019)
- [On the Consistency of Top-k Surrogate Losses](https://mlsanity.com/paper/1901.11141/) (2019)
- [Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination](https://mlsanity.com/paper/2108.03210/) (2021)
- [Forecast Evaluation of Quantiles, Prediction Intervals, and other Set-Valued Functionals](https://mlsanity.com/paper/1910.07912/) (2019)

## Built on by (most cited first, 5 of 5)

- [All Thresholds Barred: Direct Estimation of Call Density in Bioacoustic Data](https://mlsanity.com/paper/2402.15360/) (2024)
- [Superior Scoring Rules for Probabilistic Evaluation of Single-Label Multi-Class Classification Tasks](https://mlsanity.com/paper/2407.17697/) (2024)
- [Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search](https://mlsanity.com/paper/2605.01936/) (2026)
- [Calibrated Preference Learning: The Case of Label Ranking](https://mlsanity.com/paper/2605.30447/) (2026)
- [Loss-Shift Transfer via Bayes Quotients](https://mlsanity.com/paper/2606.13178/) (2026)

## Abstract (arXiv)

In the face of uncertainty, the need for probabilistic assessments has long been recognized in the literature on forecasting. In classification, however, comparative evaluation of classifiers often focuses on predictions specifying a single class through the use of simple accuracy measures, which disregard any probabilistic uncertainty quantification. I propose probabilistic top lists as a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions. The probabilistic top list functional is elicitable through the use of strictly consistent evaluation metrics. The proposed evaluation metrics are based on symmetric proper scoring rules and admit comparison of various types of predictions ranging from single-class point predictions to fully specified predictive distributions. The Brier score yields a metric that is particularly well suited for this kind of comparison.