# Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification

arXiv: https://arxiv.org/abs/2408.10193 · HTML: https://mlsanity.com/paper/2408.10193/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 50 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 3 of 3)

- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters](https://mlsanity.com/paper/2003.01182/) (2020)
- [Limitations of ROC on Imbalanced Data: Evaluation of LVAD Mortality Risk Scores](https://mlsanity.com/paper/2010.16253/) (2020)

## Built on by (most cited first, 7 of 7)

- [From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions](https://mlsanity.com/paper/2603.13359/) (2026)
- [Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Bias](https://mlsanity.com/paper/2608.11985/) (2026)
- [How Much Noise Can BERT Handle? Insights from Multilingual Sentence Difficulty Detection](https://mlsanity.com/paper/2603.07346/) (2026)
- [Frequency-Corrupt Based Graph Self-Supervised Learning](https://mlsanity.com/paper/2604.15699/) (2026)
- [Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals](https://mlsanity.com/paper/2605.05025/) (2026)
- [A Benchmark for Early-stage Parkinson's Disease Detection from Speech](https://mlsanity.com/paper/2605.14066/) (2026)
- [SAGE: A sampling-aware global evaluation benchmark for species distribution modeling](https://mlsanity.com/paper/2609.31082/) (2026)

## Abstract (arXiv)

The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables and the sample size are kept constant. Analyzing 156 data scenarios, 18 model evaluation metrics and five commonly used machine learning models as well as a naive random guess model, I find that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, Area Under the ROC Curve (AUC) which takes all decision thresholds into account when evaluating models has the smallest variance in evaluating individual models and smallest variance in ranking of a set of models. A close threshold analysis using all possible thresholds for all metrics further supports the hypothesis that considering all decision thresholds helps reduce the variance in model evaluation with respect to prevalence change in data. The results have significant implications for model evaluation and model selection in binary classification tasks.