# JAWS: Auditing Predictive Uncertainty Under Covariate Shift

arXiv: https://arxiv.org/abs/2207.10716 · HTML: https://mlsanity.com/paper/2207.10716/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 78 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Conformal Prediction: a Unified Review of Theory and New Challenges](https://mlsanity.com/paper/2005.07972/) (2020)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Robust Fairness under Covariate Shift](https://mlsanity.com/paper/2010.05166/) (2020)
- [A Swiss Army Infinitesimal Jackknife](https://mlsanity.com/paper/1806.00550/) (2018)
- [Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data](https://mlsanity.com/paper/2011.03274/) (2020)
- [Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions](https://mlsanity.com/paper/2007.13481/) (2020)
- [Can You Trust This Prediction? Auditing Pointwise Reliability After Learning](https://mlsanity.com/paper/1901.00403/) (2019)
- [Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI](https://mlsanity.com/paper/2106.01410/) (2021)
- [Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift](https://mlsanity.com/paper/2006.14988/) (2020)
- [Computationally efficient versions of conformal predictive distributions](https://mlsanity.com/paper/1911.00941/) (2019)
- [Computing Full Conformal Prediction Set with Approximate Homotopy](https://mlsanity.com/paper/1909.09365/) (2019)
- [Robust Importance Weighting for Covariate Shift](https://mlsanity.com/paper/1910.06324/) (2019)
- [A Higher-Order Swiss Army Infinitesimal Jackknife](https://mlsanity.com/paper/1907.12116/) (2019)
- [Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment](https://mlsanity.com/paper/2110.06448/) (2021)

## Built on by (most cited first, 14 of 14)

- [Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)](https://mlsanity.com/paper/2405.06627/) (2024)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Training-Conditional Coverage Bounds under Covariate Shift](https://mlsanity.com/paper/2405.16594/) (2024)
- [Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shifts](https://mlsanity.com/paper/2403.05600/) (2024)
- [Robust Conformal Prediction Using Privileged Information](https://mlsanity.com/paper/2406.05405/) (2024)
- [Distribution-free risk assessment of regression-based machine learning algorithms](https://mlsanity.com/paper/2310.03545/) (2023)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Conformal Predictive Systems Under Covariate Shift](https://mlsanity.com/paper/2404.15018/) (2024)
- [Improving the statistical efficiency of cross-conformal prediction](https://mlsanity.com/paper/2503.01495/) (2025)
- [Calibrating Decision Robustness via Inverse Conformal Risk Control](https://mlsanity.com/paper/2510.07750/) (2025)
- [Conformalized Decision Risk Assessment](https://mlsanity.com/paper/2505.13243/) (2025)
- [Sharp training-conditional coverage for conformal prediction under covariate shift](https://mlsanity.com/paper/2609.33456/) (2026)
- [Distribution-Free Predictive Inference under Unknown Temporal Drift](https://mlsanity.com/paper/2406.06516/) (2024)
- [An In-Depth Examination of Risk Assessment in Multi-Class Classification Algorithms](https://mlsanity.com/paper/2412.04166/) (2024)

## Abstract (arXiv)

We propose JAWS, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method JAW, the JAckknife+ Weighted with data-dependent likelihood-ratio weights. JAWS also includes computationally efficient Approximations of JAW using higher-order influence functions: JAWA. Theoretically, we show that JAW relaxes the jackknife+'s assumption of data exchangeability to achieve the same finite-sample coverage guarantee even under covariate shift. JAWA further approaches the JAW guarantee in the limit of the sample size or the influence function order under common regularity assumptions. Moreover, we propose a general approach to repurposing predictive interval-generating methods and their guarantees to the reverse task: estimating the probability that a prediction is erroneous, based on user-specified error criteria such as a safe or acceptable tolerance threshold around the true label. We then propose JAW-E and JAWA-E as the repurposed proposed methods for this Error assessment task. Practically, JAWS outperform state-of-the-art predictive inference baselines in a variety of biased real world data sets for interval-generation and error-assessment predictive uncertainty auditing tasks.