# Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration

arXiv: https://arxiv.org/abs/2503.18025 · HTML: https://mlsanity.com/paper/2503.18025/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 56 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [High-Dimensional Prediction for Sequential Decision Making](https://mlsanity.com/paper/2310.17651/) (2023)
- [Meta-Cal: Well-controlled Post-hoc Calibration by Ranking](https://mlsanity.com/paper/2105.04290/) (2021)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare](https://mlsanity.com/paper/2202.01906/) (2022)
- [Cost-aware Pre-training for Multiclass Cost-sensitive Deep Learning](https://mlsanity.com/paper/1511.09337/) (2015)
- [Decision-Making under Miscalibration](https://mlsanity.com/paper/2203.09852/) (2022)

## Built on by (most cited first, 3 of 3)

- [Role Support in Knowledge-Graph Error Ranking: Predictor Regimes and Evaluation Policies](https://mlsanity.com/paper/2512.22318/) (2025)
- [On the calibration of survival models with competing risks](https://mlsanity.com/paper/2602.00194/) (2026)
- [Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference](https://mlsanity.com/paper/2606.03332/) (2026)

## Abstract (arXiv)

Probabilistic classifiers are central for making informed decisions under uncertainty. Based on the maximum expected utility principle, optimal decision rules can be derived using the posterior class probabilities and misclassification costs. Yet, in practice only learned approximations of the oracle posterior probabilities are available. In this work, we quantify the excess risk (a.k.a. regret) incurred using approximate posterior probabilities in batch binary decision-making. We provide analytical expressions for miscalibration-induced regret (RCL), as well as tight and informative upper and lower bounds on the regret of calibrated classifiers (RGL). These expressions allow us to identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss, which calls for post-training beyond recalibration. Crucially, both RCL and RGL can be estimated in practice using a calibration curve and a recent grouping loss estimator. On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost. Finally, we highlight the potential of multicalibration approaches as efficient alternatives to costlier fine-tuning approaches.