# On the good reliability of an interval-based metric to validate prediction uncertainty for machine learning regression tasks

arXiv: https://arxiv.org/abs/2408.13089 · HTML: https://mlsanity.com/paper/2408.13089/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 11 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 1 of 1)

- [Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity](https://mlsanity.com/paper/2309.06240/) (2023)

## Abstract (arXiv)

This short study presents an opportunistic approach to a (more) reliable validation method for prediction uncertainty average calibration. Considering that variance-based calibration metrics (ZMS, NLL, RCE...) are quite sensitive to the presence of heavy tails in the uncertainty and error distributions, a shift is proposed to an interval-based metric, the Prediction Interval Coverage Probability (PICP). It is shown on a large ensemble of molecular properties datasets that (1) sets of z-scores are well represented by Student's-t(nu) distributions, nu being the number of degrees of freedom; (2) accurate estimation of 95 % prediction intervals can be obtained by the simple 2σ rule for nu>3; and (3) the resulting PICPs are more quickly and reliably tested than variance-based calibration metrics. Overall, this method enables to test 20 % more datasets than ZMS testing. Conditional calibration is also assessed using the PICP approach.