# Cost-Sensitive Evaluation for Binary Classifiers

arXiv: https://arxiv.org/abs/2510.22016 · HTML: https://mlsanity.com/paper/2510.22016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 43 references that are in the map.
- First posted on arXiv 24 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning](https://mlsanity.com/paper/1609.06570/) (2016)
- [Extending F1 metric, probabilistic approach](https://mlsanity.com/paper/2210.11997/) (2022)
- [Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models](https://mlsanity.com/paper/2010.04486/) (2020)
- [Metric Learning from Imbalanced Data](https://mlsanity.com/paper/1909.01651/) (2019)
- [Is F1 Score Suboptimal for Cybersecurity Models? Introducing Cscore, a Cost-Aware Alternative for Model Assessment](https://mlsanity.com/paper/2407.14664/) (2024)
- [Standardization of Weighted Ranking Correlation Coefficients](https://mlsanity.com/paper/2504.08428/) (2025)

## Abstract (arXiv)

Selecting an appropriate evaluation metric for classifiers is crucial for model comparison, parameter optimization, and deployment decisions, yet there is no consensus on a broadly accepted evaluation paradigm explicitly aligned with Total Classification Cost (TCC) minimization. At the same time, class imbalance is often treated as a problem to be corrected per se, potentially causing misalignments with TCC minimization. To address these limitations, (i) we define Weighted Accuracy (WA), an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of accuracy and (ii) we propose a general reweighting framework for handling class imbalance in cost-sensitive scenarios, providing an alternative to resampling techniques. This framework applies to any evaluation metric or loss function that can be expressed as a linear combination of example-dependent quantities; it enables meaningful comparison of evaluation results obtained on different datasets and accounts for discrepancies between the development dataset, used for training, validation, and testing, and the target dataset, where the model will be deployed. Within this framework, we derive the conditions under which standard rebalancing techniques remain coherent with TCC minimization, and when they may instead become misleading. We prove that, under example-independent Unit Classification Costs, maximizing WA is equivalent to minimizing TCC. Finally, we analyze the robustness of WA in realistic example-dependent cost scenarios by studying its correlation with TCC across a broad range of class imbalance and cost regimes. The results show that WA maintains robust alignment with TCC across almost all examined scenarios.