# Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics

arXiv: https://arxiv.org/abs/2605.01796 · HTML: https://mlsanity.com/paper/2605.01796/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 3 May 2026.

## Builds on (most cited first, 11 of 11)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Metrics of calibration for probabilistic predictions](https://mlsanity.com/paper/2205.09680/) (2022)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)
- [On the Distance from Calibration in Sequential Prediction](https://mlsanity.com/paper/2402.07458/) (2024)
- [How Flawed Is ECE? An Analysis via Logit Smoothing](https://mlsanity.com/paper/2402.10046/) (2024)
- [TCE: A Test-Based Approach to Measuring Calibration Error](https://mlsanity.com/paper/2306.14343/) (2023)
- [An Entropic Metric for Measuring Calibration of Machine Learning Models](https://mlsanity.com/paper/2502.14545/) (2025)
- [Trust, or Don't Predict: Introducing the CWSA Family for Confidence-Aware Model Evaluation](https://mlsanity.com/paper/2505.18622/) (2025)

## Built on by (most cited first, 1 of 1)

- [Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift](https://mlsanity.com/paper/2606.22659/) (2026)

## Abstract (arXiv)

Confidence calibration has been dominated by the Expected Calibration Error (ECE), a linear metric that counts calibration offset equally regardless of the confidence level at which it occurs. We show that ECE can remain small even under arbitrarily large overconfidence risk, so we propose Calibrated Size Ratio (CSR) instead, an interpretable metric that equals 1 under perfect calibration, from which we derive the risk probability Prisk that quantifies the statistical evidence for overconfidence. We further argue that overconfidence risk assessment must be complemented by a measure of discriminative value: whether the assigned confidences actively distinguish correct from incorrect predictions. We show that confidence-weighted accuracy cwA is the natural such complement, and that confidence-weighting extends to all standard classification metrics. In particular, we prove that the confidence-weighted AUC (cwAUC) captures the information about calibration while the classical AUC cannot. We validate the proposed indicators on several synthetic confidence distributions under multiple controlled calibration profiles and find that CSR separates risky from non-risky assignments. We also test the metrics on fifteen real datasets, with and without post-hoc calibration, and find that standard methods can yield risky confidence profiles.