# Properties of the ENCE and other MAD-based calibration metrics

arXiv: https://arxiv.org/abs/2305.11905 · HTML: https://mlsanity.com/paper/2305.11905/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 5 of 5)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks](https://mlsanity.com/paper/2107.06068/) (2021)
- [Uncertainty quantification for predictions of atomistic neural networks](https://mlsanity.com/paper/2207.06916/) (2022)

## Built on by (most cited first, 7 of 7)

- [Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity](https://mlsanity.com/paper/2309.06240/) (2023)
- [Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics](https://mlsanity.com/paper/2306.05180/) (2023)
- [Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?](https://mlsanity.com/paper/2310.11978/) (2023)
- [Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks](https://mlsanity.com/paper/2402.10043/) (2024)
- [On Uncertainty Calibration for Equivariant Functions](https://mlsanity.com/paper/2510.21691/) (2025)
- [A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis](https://mlsanity.com/paper/2511.00301/) (2025)
- [Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysis](https://mlsanity.com/paper/2403.00423/) (2024)

## Abstract (arXiv)

The Expected Normalized Calibration Error (ENCE) is a popular calibration statistic used in Machine Learning to assess the quality of prediction uncertainties for regression problems. Estimation of the ENCE is based on the binning of calibration data. In this short note, I illustrate an annoying property of the ENCE, i.e. its proportionality to the square root of the number of bins for well calibrated or nearly calibrated datasets. A similar behavior affects the calibration error based on the variance of z-scores (ZVE), and in both cases this property is a consequence of the use of a Mean Absolute Deviation (MAD) statistic to estimate calibration errors. Hence, the question arises of which number of bins to choose for a reliable estimation of calibration error statistics. A solution is proposed to infer ENCE and ZVE values that do not depend on the number of bins for datasets assumed to be calibrated, providing simultaneously a statistical calibration test. It is also shown that the ZVE is less sensitive than the ENCE to outstanding errors or uncertainties.