# Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

arXiv: https://arxiv.org/abs/2010.03039 · HTML: https://mlsanity.com/paper/2010.03039/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [A review of predictive uncertainty estimation with machine learning](https://mlsanity.com/paper/2209.08307/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [Locally Valid and Discriminative Prediction Intervals for Deep Learning Models](https://mlsanity.com/paper/2106.00225/) (2021)
- [Interpretable Uncertainty Quantification in AI for HEP](https://mlsanity.com/paper/2208.03284/) (2022)
- [Learning-To-Measure: In-Context Active Feature Acquisition](https://mlsanity.com/paper/2510.12624/) (2025)
- [CP2: Leveraging Geometry for Conformal Prediction via Canonicalization](https://mlsanity.com/paper/2506.16189/) (2025)
- [Uncertainty Quantification for Image-based Traffic Prediction across Cities](https://mlsanity.com/paper/2308.06129/) (2023)
- [On Equivariant Model Selection through the Lens of Uncertainty](https://mlsanity.com/paper/2506.18629/) (2025)
- [Scalable computation of prediction intervals for neural networks via matrix sketching](https://mlsanity.com/paper/2205.03194/) (2022)
- [Confidence-Nets: A Step Towards better Prediction Intervals for regression Neural Networks on small datasets](https://mlsanity.com/paper/2210.17092/) (2022)
- [Comparing the quality of neural network uncertainty estimates for classification problems](https://mlsanity.com/paper/2308.05903/) (2023)
- [On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning](https://mlsanity.com/paper/2311.12688/) (2023)
- [Robust and Reliable AI for Predictive Quality in Semiconductor Materials Manufacturing with MLOps and Uncertainty Quantification](https://mlsanity.com/paper/2605.07752/) (2026)
- [Uncertainty-aware classification and triage of structural heart disease using electrocardiography and echocardiography metrics](https://mlsanity.com/paper/2605.22968/) (2026)
- [Uncertainty-aware reinforcement learning for chemical language models](https://mlsanity.com/paper/2606.24990/) (2026)

## Abstract (arXiv)

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated using point-prediction metrics such as negative log-likelihood or the Brier score on heldout data. In this study, we provide the first large scale evaluation of the empirical frequentist coverage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and establish coverage as an important metric in developing models for real-world applications.