# Never mind the metrics -- what about the uncertainty? Visualising confusion matrix metric distributions

arXiv: https://arxiv.org/abs/2206.02157 · HTML: https://mlsanity.com/paper/2206.02157/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 49 references that are in the map.
- First posted on arXiv 5 Jun 2022.

## Builds on (most cited first, 6 of 6)

- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Good Classification Measures and How to Find Them](https://mlsanity.com/paper/2201.09044/) (2022)
- [Prevalence Threshold and the Geometry of Screening Curves](https://mlsanity.com/paper/2006.00398/) (2020)
- [Appropriateness of Performance Indices for Imbalanced Data Classification: An Analysis](https://mlsanity.com/paper/2008.11752/) (2020)
- [Classifier uncertainty: evidence, potential impact, and probabilistic treatment](https://mlsanity.com/paper/2006.11105/) (2020)

## Abstract (arXiv)

There are strong incentives to build models that demonstrate outstanding predictive performance on various datasets and benchmarks. We believe these incentives risk a narrow focus on models and on the performance metrics used to evaluate and compare them -- resulting in a growing body of literature to evaluate and compare metrics. This paper strives for a more balanced perspective on classifier performance metrics by highlighting their distributions under different models of uncertainty and showing how this uncertainty can easily eclipse differences in the empirical performance of classifiers. We begin by emphasising the fundamentally discrete nature of empirical confusion matrices and show how binary matrices can be meaningfully represented in a three dimensional compositional lattice, whose cross-sections form the basis of the space of receiver operating characteristic (ROC) curves. We develop equations, animations and interactive visualisations of the contours of performance metrics within (and beyond) this ROC space, showing how some are affected by class imbalance. We provide interactive visualisations that show the discrete posterior predictive probability mass functions of true and false positive rates in ROC space, and how these relate to uncertainty in performance metrics such as Balanced Accuracy (BA) and the Matthews Correlation Coefficient (MCC). Our hope is that these insights and visualisations will raise greater awareness of the substantial uncertainty in performance metric estimates that can arise when classifiers are evaluated on empirical datasets and benchmarks, and that classification model performance claims should be tempered by this understanding.