# Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints

arXiv: https://arxiv.org/abs/2510.18520 · HTML: https://mlsanity.com/paper/2510.18520/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 58 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [Scalable Learning of Non-Decomposable Objectives](https://mlsanity.com/paper/1608.04802/) (2016)
- [An Extensive Data Processing Pipeline for MIMIC-IV](https://mlsanity.com/paper/2204.13841/) (2022)
- [Robust and Decomposable Average Precision for Image Retrieval](https://mlsanity.com/paper/2110.01445/) (2021)
- [AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning](https://mlsanity.com/paper/1912.00965/) (2019)
- [Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers](https://mlsanity.com/paper/2009.01367/) (2020)
- [Exact Reformulation and Optimization for Direct Metric Optimization in Binary Imbalanced Classification](https://mlsanity.com/paper/2507.15240/) (2025)

## Abstract (arXiv)

The ROC curve is widely used to assess binary classifiers. Yet for some applications, such as alert systems for monitoring hospitalized patients, conventional ROC analysis cannot meet two key deployment needs: enforcing a constraint on precision to avoid false alarm fatigue and imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet precision and capacity constraints occupy a feasible region in ROC space. We establish the polygon-shaped geometry of this region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible region. Averaging this area over a desired distribution for cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk from vital sign history on several datasets, we show this cost-aware metric can outperform alternatives at ranking classifiers for in-hospital alerts.