# A Ranking Approach for Measuring Calibration

arXiv: https://arxiv.org/abs/2609.13100 · HTML: https://mlsanity.com/paper/2609.13100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2026.

## Abstract (arXiv)

When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that Y=1) exactly matches the forecasted probability f(X). In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability f(X). Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.