# Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions

arXiv: https://arxiv.org/abs/2007.13481 · HTML: https://mlsanity.com/paper/2007.13481/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 30 of 34)

- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Post-hoc Uncertainty Learning using a Dirichlet Meta-Model](https://mlsanity.com/paper/2212.07359/) (2022)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [Locally Valid and Discriminative Prediction Intervals for Deep Learning Models](https://mlsanity.com/paper/2106.00225/) (2021)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [Stable Conformal Prediction Sets](https://mlsanity.com/paper/2112.10224/) (2021)
- [Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?](https://mlsanity.com/paper/2402.06160/) (2024)
- [Learning Prediction Intervals for Regression: Generalization and Calibration](https://mlsanity.com/paper/2102.13625/) (2021)
- [JuryGCN: Quantifying Jackknife Uncertainty on Graph Convolutional Networks](https://mlsanity.com/paper/2210.05959/) (2022)
- [JAWS: Auditing Predictive Uncertainty Under Covariate Shift](https://mlsanity.com/paper/2207.10716/) (2022)
- [Data-SUITE: Data-centric identification of in-distribution incongruous examples](https://mlsanity.com/paper/2202.08836/) (2022)
- [Approximating Full Conformal Prediction at Scale via Influence Functions](https://mlsanity.com/paper/2202.01315/) (2022)
- [Generalization Through The Lens Of Leave-One-Out Error](https://mlsanity.com/paper/2203.03443/) (2022)
- [Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models](https://mlsanity.com/paper/2405.03869/) (2024)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [Towards Reliable Rare Category Analysis on Graphs via Individual Calibration](https://mlsanity.com/paper/2307.09858/) (2023)
- [Adaptive Instrument Design for Indirect Experiments](https://mlsanity.com/paper/2312.02438/) (2023)
- [Extreme Conformal Prediction: Reliable Intervals for High-Impact Events](https://mlsanity.com/paper/2505.08578/) (2025)
- [A Free Lunch with Influence Functions? Improving Neural Network Estimates with Concepts from Semiparametric Statistics](https://mlsanity.com/paper/2202.09096/) (2022)
- [A Survey on Uncertainty Toolkits for Deep Learning](https://mlsanity.com/paper/2205.01040/) (2022)
- [Generative vs. Discriminative modeling under the lens of uncertainty quantification](https://mlsanity.com/paper/2406.09172/) (2024)
- [Confidence Intervals and Simultaneous Confidence Bands Based on Deep Learning](https://mlsanity.com/paper/2406.14009/) (2024)
- [Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence](https://mlsanity.com/paper/2507.20272/) (2025)
- [IMU: Influence-guided Machine Unlearning](https://mlsanity.com/paper/2508.01620/) (2025)
- [Uncertainty-Aware Prediction of Parkinson's Disease Medication Needs: A Two-Stage Conformal Prediction Approach](https://mlsanity.com/paper/2508.10284/) (2025)
- [PARIS: Pruning Algorithm via the Representer theorem for Imbalanced Scenarios](https://mlsanity.com/paper/2512.06950/) (2025)
- [Safe AI for health and beyond -- Monitoring to transform a health service](https://mlsanity.com/paper/2303.01513/) (2023)
- [Probabilistic Shapley Value Modeling and Inference](https://mlsanity.com/paper/2402.04211/) (2024)

## Abstract (arXiv)

Deep learning models achieve high predictive accuracy across a broad spectrum of tasks, but rigorously quantifying their predictive uncertainty remains challenging. Usable estimates of predictive uncertainty should (1) cover the true prediction targets with high probability, and (2) discriminate between high- and low-confidence prediction instances. Existing methods for uncertainty quantification are based predominantly on Bayesian neural networks; these may fall short of (1) and (2) -- i.e., Bayesian credible intervals do not guarantee frequentist coverage, and approximate posterior inference undermines discriminative accuracy. In this paper, we develop the discriminative jackknife (DJ), a frequentist procedure that utilizes influence functions of a model's loss functional to construct a jackknife (or leave-one-out) estimator of predictive confidence intervals. The DJ satisfies (1) and (2), is applicable to a wide range of deep learning models, is easy to implement, and can be applied in a post-hoc fashion without interfering with model training or compromising its accuracy. Experiments demonstrate that DJ performs competitively compared to existing Bayesian and non-Bayesian regression baselines.