# PAC Confidence Predictions for Deep Neural Network Classifiers

arXiv: https://arxiv.org/abs/2011.00716 · HTML: https://mlsanity.com/paper/2011.00716/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 21 of 21)

- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [PAC Prediction Sets Under Covariate Shift](https://mlsanity.com/paper/2106.09848/) (2021)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [PAC Prediction Sets Under Label Shift](https://mlsanity.com/paper/2310.12964/) (2023)
- [Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs](https://mlsanity.com/paper/2402.08733/) (2024)
- [Distributionally Robust Statistical Verification with Imprecise Neural Networks](https://mlsanity.com/paper/2308.14815/) (2023)
- [Towards PAC Multi-Object Detection and Tracking](https://mlsanity.com/paper/2204.07482/) (2022)
- [Confidence Composition for Monitors of Verification Assumptions](https://mlsanity.com/paper/2111.03782/) (2021)
- [Efficient and Multiply Robust Risk Estimation under General Forms of Dataset Shift](https://mlsanity.com/paper/2306.16406/) (2023)
- [Distribution-Free Finite-Sample Guarantees and Split Conformal Prediction](https://mlsanity.com/paper/2210.14735/) (2022)
- [Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction](https://mlsanity.com/paper/2405.15912/) (2024)
- [Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking](https://mlsanity.com/paper/2205.09638/) (2022)
- [Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction](https://mlsanity.com/paper/2403.19605/) (2024)
- [Cautious Calibration in Binary Classification](https://mlsanity.com/paper/2408.05120/) (2024)
- [Confidence Calibration with Bounded Error Using Transformations](https://mlsanity.com/paper/2102.12680/) (2021)
- [Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching](https://mlsanity.com/paper/2110.05390/) (2021)
- [Distribution-Free Predictive Inference under Unknown Temporal Drift](https://mlsanity.com/paper/2406.06516/) (2024)
- [Position: There Is No Free Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2506.03670/) (2025)

## Abstract (arXiv)

A key challenge for deploying deep neural networks (DNNs) in safety critical settings is the need to provide rigorous ways to quantify their uncertainty. In this paper, we propose a novel algorithm for constructing predicted classification confidences for DNNs that comes with provable correctness guarantees. Our approach uses Clopper-Pearson confidence intervals for the Binomial distribution in conjunction with the histogram binning approach to calibrated prediction. In addition, we demonstrate how our predicted confidences can be used to enable downstream guarantees in two settings: (i) fast DNN inference, where we demonstrate how to compose a fast but inaccurate DNN with an accurate but slow DNN in a rigorous way to improve performance without sacrificing accuracy, and (ii) safe planning, where we guarantee safety when using a DNN to predict whether a given action is safe based on visual observations. In our experiments, we demonstrate that our approach can be used to provide guarantees for state-of-the-art DNNs.