# Testing the Consistency of Performance Scores Reported for Binary Classification Problems

arXiv: https://arxiv.org/abs/2310.12527 · HTML: https://mlsanity.com/paper/2310.12527/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 75 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [AUC Maximization in the Era of Big Data and AI: A Survey](https://mlsanity.com/paper/2203.15046/) (2022)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Deep Learning Ensembles for Melanoma Recognition in Dermoscopy Images](https://mlsanity.com/paper/1610.04662/) (2016)
- [Validating Hyperspectral Image Segmentation](https://mlsanity.com/paper/1811.03707/) (2018)
- [Skin lesion detection based on an ensemble of deep convolutional neural network](https://mlsanity.com/paper/1705.03360/) (2017)
- [Prevalence Threshold and the Geometry of Screening Curves](https://mlsanity.com/paper/2006.00398/) (2020)
- [A Comprehensive Review of Modern Object Segmentation Approaches](https://mlsanity.com/paper/2301.07499/) (2023)
- [A new baseline for retinal vessel segmentation: Numerical identification and correction of methodological inconsistencies affecting 100+ papers](https://mlsanity.com/paper/2111.03853/) (2021)
- [A pipeline and comparative study of 12 machine learning models for text classification](https://mlsanity.com/paper/2204.06518/) (2022)
- [The Prevalence of Errors in Machine Learning Experiments](https://mlsanity.com/paper/1909.04436/) (2019)

## Built on by (most cited first, 1 of 1)

- [Enumerating the k-fold configurations in multi-class classification problems](https://mlsanity.com/paper/2401.13843/) (2024)

## Abstract (arXiv)

Binary classification is a fundamental task in machine learning, with applications spanning various scientific domains. Whether scientists are conducting fundamental research or refining practical applications, they typically assess and rank classification techniques based on performance metrics such as accuracy, sensitivity, and specificity. However, reported performance scores may not always serve as a reliable basis for research ranking. This can be attributed to undisclosed or unconventional practices related to cross-validation, typographical errors, and other factors. In a given experimental setup, with a specific number of positive and negative test items, most performance scores can assume specific, interrelated values. In this paper, we introduce numerical techniques to assess the consistency of reported performance scores and the assumed experimental setup. Importantly, the proposed approach does not rely on statistical inference but uses numerical methods to identify inconsistencies with certainty. Through three different applications related to medicine, we demonstrate how the proposed techniques can effectively detect inconsistencies, thereby safeguarding the integrity of research fields. To benefit the scientific community, we have made the consistency tests available in an open-source Python package.