# Performance Metrics for Probabilistic Ordinal Classifiers

arXiv: https://arxiv.org/abs/2309.08701 · HTML: https://mlsanity.com/paper/2309.08701/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 7 of 7)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detection](https://mlsanity.com/paper/2105.10490/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Analysis and Comparison of Classification Metrics](https://mlsanity.com/paper/2209.05355/) (2022)
- [Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye Fundus Images](https://mlsanity.com/paper/2010.00291/) (2020)

## Built on by (most cited first, 5 of 5)

- [Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Models](https://mlsanity.com/paper/2603.20162/) (2026)
- [Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score](https://mlsanity.com/paper/2606.24959/) (2026)
- [Multi-Rater Calibrated Segmentation Models](https://mlsanity.com/paper/2605.02437/) (2026)
- [ClinReadNet: A clinical reading-inspired network for low-dose abdominal CT image quality assessment](https://mlsanity.com/paper/2606.10372/) (2026)
- [ADABORD: a novel AdaBoost approach for ordinal classification](https://mlsanity.com/paper/2607.21003/) (2026)

## Abstract (arXiv)

Ordinal classification models assign higher penalties to predictions further away from the true class. As a result, they are appropriate for relevant diagnostic tasks like disease progression prediction or medical image grading. The consensus for assessing their categorical predictions dictates the use of distance-sensitive metrics like the Quadratic-Weighted Kappa score or the Expected Cost. However, there has been little discussion regarding how to measure performance of probabilistic predictions for ordinal classifiers. In conventional classification, common measures for probabilistic predictions are Proper Scoring Rules (PSR) like the Brier score, or Calibration Errors like the ECE, yet these are not optimal choices for ordinal classification. A PSR named Ranked Probability Score (RPS), widely popular in the forecasting field, is more suitable for this task, but it has received no attention in the image analysis community. This paper advocates the use of the RPS for image grading tasks. In addition, we demonstrate a counter-intuitive and questionable behavior of this score, and propose a simple fix for it. Comprehensive experiments on four large-scale biomedical image grading problems over three different datasets show that the RPS is a more suitable performance metric for probabilistic ordinal predictions. Code to reproduce our experiments can be found at https://github.com/agaldran/prob_ord_metrics .