# A Consequentialist Critique of Binary Classification Evaluation: Theory, Practice, and Tools

arXiv: https://arxiv.org/abs/2504.04528 · HTML: https://mlsanity.com/paper/2504.04528/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 66 references that are in the map.
- First posted on arXiv 6 Apr 2025.

## Builds on (most cited first, 6 of 6)

- [The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric](https://mlsanity.com/paper/1902.05826/) (2019)
- [Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation](https://mlsanity.com/paper/2103.11357/) (2021)
- [Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare](https://mlsanity.com/paper/2202.01906/) (2022)
- [The Misuse of AUC: What High Impact Risk Assessment Gets Wrong](https://mlsanity.com/paper/2305.18159/) (2023)
- [Evaluating Probabilistic Classifiers: The Triptych](https://mlsanity.com/paper/2301.10803/) (2023)
- [Weighted Brier Score -- an Overall Summary Measure for Risk Prediction Models with Clinical Utility Consideration](https://mlsanity.com/paper/2408.01626/) (2024)

## Built on by (most cited first, 1 of 1)

- [Ensemble Diversity Optimization for Subjective Supervision](https://mlsanity.com/paper/2607.08493/) (2026)

## Abstract (arXiv)

Machine learning-supported decisions, such as ordering diagnostic tests or determining preventive custody, often require converting probabilistic forecasts into binary classifications. We adopt a consequentialist perspective from decision theory to argue that evaluation methods should prioritize forecast quality across thresholds and base rates. This motivates the use of proper scoring rules such as the Brier score and log loss. However, our empirical review of practices at major ML venues (ICML, FAccT, CHIL) reveals a dominant reliance on top-K metrics or fixed-threshold evaluations. To bridge this disconnect, we introduce a decision-theoretic framework that maps evaluation metrics to their appropriate use cases, accompanied by a practical Python package, textttbriertools, which lowers the barrier to applying proper scoring rules in practice. Methodologically, we derive and implement a clipped Brier score variant that avoids full integration and better reflects bounded, interpretable threshold ranges. Theoretically, we reconcile the Brier score with decision curve analysis, directly addressing the critique of (Assel, et al. 2017) regarding the clinical utility of proper scoring rules.