# Assessment of Prediction Intervals Using Uncertainty Characteristics Curves

arXiv: https://arxiv.org/abs/2310.03158 · HTML: https://mlsanity.com/paper/2310.03158/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Quality of Uncertainty Quantification for Bayesian Neural Network Inference](https://mlsanity.com/paper/1906.09686/) (2019)
- [Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes](https://mlsanity.com/paper/1805.05396/) (2018)
- [Crowd Counting with Decomposed Uncertainty](https://mlsanity.com/paper/1903.07427/) (2019)

## Abstract (arXiv)

Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to an ad-hoc operating point, making evaluation and comparison across different studies relatively difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a null reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper defines the Uncertainty Characteristics Curve and demonstrates its utility in selected scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox.