# Locally Valid and Discriminative Prediction Intervals for Deep Learning Models

arXiv: https://arxiv.org/abs/2106.00225 · HTML: https://mlsanity.com/paper/2106.00225/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions](https://mlsanity.com/paper/2007.13481/) (2020)
- [Conformal prediction with localization](https://mlsanity.com/paper/1908.08558/) (2019)
- [Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures](https://mlsanity.com/paper/2010.03039/) (2020)
- [SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Models](https://mlsanity.com/paper/2103.03945/) (2021)

## Built on by (most cited first, 24 of 24)

- [Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction](https://mlsanity.com/paper/2509.18658/) (2025)
- [Conformal Prediction with Temporal Quantile Adjustments](https://mlsanity.com/paper/2205.09940/) (2022)
- [Conformal Prediction with Missing Values](https://mlsanity.com/paper/2306.02732/) (2023)
- [Split Localized Conformal Prediction](https://mlsanity.com/paper/2206.13092/) (2022)
- [Conformal Prediction via Regression-as-Classification](https://mlsanity.com/paper/2404.08168/) (2024)
- [Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction](https://mlsanity.com/paper/2210.03067/) (2022)
- [Improving Uncertainty Quantification of Deep Classifiers via Neighborhood Conformal Prediction: Novel Algorithm and Theoretical Analysis](https://mlsanity.com/paper/2303.10694/) (2023)
- [Adaptive Conformal Prediction by Reweighting Nonconformity Score](https://mlsanity.com/paper/2303.12695/) (2023)
- [Approximating Full Conformal Prediction at Scale via Influence Functions](https://mlsanity.com/paper/2202.01315/) (2022)
- [Normalizing Flows for Conformal Regression](https://mlsanity.com/paper/2406.03346/) (2024)
- [Weighted Aggregation of Conformity Scores for Classification](https://mlsanity.com/paper/2407.10230/) (2024)
- [Conformal Prediction Intervals with Temporal Dependence](https://mlsanity.com/paper/2205.12940/) (2022)
- [Adaptive Conformal Regression with Jackknife+ Rescaled Scores](https://mlsanity.com/paper/2305.19901/) (2023)
- [VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation](https://mlsanity.com/paper/2604.25235/) (2026)
- [Adaptive Modeling of Uncertainties for Traffic Forecasting](https://mlsanity.com/paper/2303.09273/) (2023)
- [Benchmarking Bayesian neural networks and evaluation metrics for regression tasks](https://mlsanity.com/paper/2206.06779/) (2022)
- [Likelihood Annealing: Fast Calibrated Uncertainty for Regression](https://mlsanity.com/paper/2302.11012/) (2023)
- [MetaSTNet: Multimodal Meta-learning for Cellular Traffic Conformal Prediction](https://mlsanity.com/paper/2505.21553/) (2025)
- [Federated Conditional Conformal Prediction via Generative Models](https://mlsanity.com/paper/2510.13297/) (2025)
- [Quantifying Local Model Validity using Active Learning](https://mlsanity.com/paper/2406.07474/) (2024)
- [Large width penalization for neural network-based prediction interval estimation](https://mlsanity.com/paper/2411.19181/) (2024)
- [Robust Conditional Conformal Prediction via Branched Normalizing Flow](https://mlsanity.com/paper/2605.01868/) (2026)
- [Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model](https://mlsanity.com/paper/2606.07771/) (2026)
- [Robust Conformal Consensus: Multi-Agent LLM-as-a-Judge Interval Evaluation with Conformal Prediction](https://mlsanity.com/paper/2609.06367/) (2026)

## Abstract (arXiv)

Crucial for building trust in deep learning models for critical real-world applications is efficient and theoretically sound uncertainty quantification, a task that continues to be challenging. Useful uncertainty information is expected to have two key properties: It should be valid (guaranteeing coverage) and discriminative (more uncertain when the expected risk is high). Moreover, when combined with deep learning (DL) methods, it should be scalable and affect the DL model performance minimally. Most existing Bayesian methods lack frequentist coverage guarantees and usually affect model performance. The few available frequentist methods are rarely discriminative and/or violate coverage guarantees due to unrealistic assumptions. Moreover, many methods are expensive or require substantial modifications to the base neural network. Building upon recent advances in conformal prediction [13, 33] and leveraging the classical idea of kernel regression, we propose Locally Valid and Discriminative prediction intervals (LVD), a simple, efficient, and lightweight method to construct discriminative prediction intervals (PIs) for almost any DL model. With no assumptions on the data distribution, such PIs also offer finite-sample local coverage guarantees (contrasted to the simpler marginal coverage). We empirically verify, using diverse datasets, that besides being the only locally valid method for DL, LVD also exceeds or matches the performance (including coverage rate and prediction accuracy) of existing uncertainty quantification methods, while offering additional benefits in scalability and flexibility.