# The Certainty Ratio Cρ: a novel metric for assessing the reliability of classifier predictions

arXiv: https://arxiv.org/abs/2411.01973 · HTML: https://mlsanity.com/paper/2411.01973/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 0 of its 15 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Abstract (arXiv)

Evaluating the performance of classifiers is critical in machine learning, particularly in high-stakes applications where the reliability of predictions can significantly impact decision-making. Traditional performance measures, such as accuracy and F-score, often fail to account for the uncertainty inherent in classifier predictions, leading to potentially misleading assessments. This paper introduces the Certainty Ratio (Cρ), a novel metric designed to quantify the contribution of confident (certain) versus uncertain predictions to any classification performance measure. By integrating the Probabilistic Confusion Matrix (CM^star) and decomposing predictions into certainty and uncertainty components, Cρ provides a more comprehensive evaluation of classifier reliability. Experimental results across 21 datasets and multiple classifiers, including Decision Trees, Naive-Bayes, 3-Nearest Neighbors, and Random Forests, demonstrate that Cρ reveals critical insights that conventional metrics often overlook. These findings emphasize the importance of incorporating probabilistic information into classifier evaluation, offering a robust tool for researchers and practitioners seeking to improve model trustworthiness in complex environments.