# Analysis and Comparison of Classification Metrics

arXiv: https://arxiv.org/abs/2209.05355 · HTML: https://mlsanity.com/paper/2209.05355/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 9 of 9)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation](https://mlsanity.com/paper/2112.08594/) (2021)
- [Binary Classifier Calibration: Non-parametric approach](https://mlsanity.com/paper/1401.3390/) (2014)
- [Likelihood-ratio calibration using prior-weighted proper scoring rules](https://mlsanity.com/paper/1307.7981/) (2013)
- [Out of a hundred trials, how many errors does your speaker verifier make?](https://mlsanity.com/paper/2104.00732/) (2021)
- [Generative Modelling for Unsupervised Score Calibration](https://mlsanity.com/paper/1311.0707/) (2013)

## Built on by (most cited first, 16 of 16)

- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Multi-Head Multi-Loss Model Calibration](https://mlsanity.com/paper/2303.01099/) (2023)
- [Towards unraveling calibration biases in medical image analysis](https://mlsanity.com/paper/2305.05101/) (2023)
- [Performance Metrics for Probabilistic Ordinal Classifiers](https://mlsanity.com/paper/2309.08701/) (2023)
- [Deployment of Image Analysis Algorithms under Prevalence Shifts](https://mlsanity.com/paper/2303.12540/) (2023)
- [A metrological framework for uncertainty evaluation in machine learning classification models](https://mlsanity.com/paper/2504.03359/) (2025)
- [Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs](https://mlsanity.com/paper/2506.14540/) (2025)
- [Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection](https://mlsanity.com/paper/2502.04528/) (2025)
- [Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models](https://mlsanity.com/paper/2307.06713/) (2023)
- [Evaluating AI capabilities in detecting conspiracy theories on YouTube](https://mlsanity.com/paper/2505.23570/) (2025)
- [Mispronunciation detection using self-supervised speech representations](https://mlsanity.com/paper/2307.16324/) (2023)
- [Deep Learning Models for Classification of COVID-19 Cases by Medical Images](https://mlsanity.com/paper/2310.16851/) (2023)
- [Time-dependent density estimation using binary classifiers](https://mlsanity.com/paper/2506.15505/) (2025)
- [hYOLO Model: Enhancing Object Classification with Hierarchical Context in YOLOv8](https://mlsanity.com/paper/2510.23278/) (2025)
- [LxCIM: a new rank-based binary classifier performance metric invariant to local exchange of classes](https://mlsanity.com/paper/2512.10053/) (2025)
- [ClinReadNet: A clinical reading-inspired network for low-dose abdominal CT image quality assessment](https://mlsanity.com/paper/2606.10372/) (2026)

## Abstract (arXiv)

A variety of different performance metrics are commonly used in the machine learning literature for the evaluation of classification systems. Some of the most common ones for measuring quality of hard decisions are standard and balanced accuracy, standard and balanced error rate, F-beta score, and Matthews correlation coefficient (MCC). In this document, we review the definition of these and other metrics and compare them with the expected cost (EC), a metric introduced in every statistical learning course but rarely used in the machine learning literature. We show that both the standard and balanced error rates are special cases of the EC. Further, we show its relation with F-beta score and MCC and argue that EC is superior to these traditional metrics for being based on first principles from statistics, and for being more general, interpretable, and adaptable to any application scenario. The metrics mentioned above measure the quality of hard decisions. Yet, most modern classification systems output continuous scores for the classes which we may want to evaluate directly. Metrics for measuring the quality of system scores include the area under the ROC curve, equal error rate, cross-entropy, Brier score, and Bayes EC or Bayes risk, among others. The last three metrics are special cases of a family of metrics given by the expected value of proper scoring rules (PSRs). We review the theory behind these metrics, showing that they are a principled way to measure the quality of the posterior probabilities produced by a system. Finally, we show how to use these metrics to compute a system's calibration loss and compare this metric with the widely-used expected calibration error (ECE), arguing that calibration loss based on PSRs is superior to the ECE for being more interpretable, more general, and directly applicable to the multi-class case, among other reasons.