# Performance measures for classification systems with rejection

arXiv: https://arxiv.org/abs/1504.02763 · HTML: https://mlsanity.com/paper/1504.02763/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2015.

## Built on by (most cited first, 16 of 16)

- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [Overcoming Common Flaws in the Evaluation of Selective Classification Systems](https://mlsanity.com/paper/2407.01032/) (2024)
- [An Impartial Take to the CNN vs Transformer Robustness Contest](https://mlsanity.com/paper/2207.11347/) (2022)
- [Explainability through uncertainty: Trustworthy decision-making with neural networks](https://mlsanity.com/paper/2403.10168/) (2024)
- [Predictable Artificial Intelligence](https://mlsanity.com/paper/2310.06167/) (2023)
- [On the use of uncertainty in classifying Aedes Albopictus mosquitoes](https://mlsanity.com/paper/2110.15912/) (2021)
- [Finding Competence Regions in Domain Generalization](https://mlsanity.com/paper/2303.09989/) (2023)
- [Safety Monitoring of Machine Learning Perception Functions: a Survey](https://mlsanity.com/paper/2412.06869/) (2024)
- [Counterfactually Comparing Abstaining Classifiers](https://mlsanity.com/paper/2305.10564/) (2023)
- [Buffer-free Class-Incremental Learning with Out-of-Distribution Detection](https://mlsanity.com/paper/2505.23412/) (2025)
- [Precision and Recall Reject Curves for Classification](https://mlsanity.com/paper/2308.08381/) (2023)
- [Trinary Tools for Continuously Valued Binary Classifiers](https://mlsanity.com/paper/2204.08136/) (2022)
- [Offline Policy Comparison with Confidence: Benchmarks and Baselines](https://mlsanity.com/paper/2205.10739/) (2022)
- [A Meta-heuristic Approach to Estimate and Explain Classifier Uncertainty](https://mlsanity.com/paper/2304.10284/) (2023)
- [Structured Prediction with Abstention via the Lov'asz Hinge](https://mlsanity.com/paper/2505.06446/) (2025)
- [A Unified Algebraic Framework for Classification Performance Evaluation](https://mlsanity.com/paper/2607.04028/) (2026)

## Abstract (arXiv)

Classifiers with rejection are essential in real-world applications where misclassifications and their effects are critical. However, if no problem specific cost function is defined, there are no established measures to assess the performance of such classifiers. We introduce a set of desired properties for performance measures for classifiers with rejection, based on which we propose a set of three performance measures for the evaluation of the performance of classifiers with rejection that satisfy the desired properties. The nonrejected accuracy measures the ability of the classifier to accurately classify nonrejected samples; the classification quality measures the correct decision making of the classifier with rejector; and the rejection quality measures the ability to concentrate all misclassified samples onto the set of rejected samples. From the measures, we derive the concept of relative optimality that allows us to connect the measures to a family of cost functions that take into account the trade-off between rejection and misclassification. We illustrate the use of the proposed performance measures on classifiers with rejection applied to synthetic and real-world data.