# PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

arXiv: https://arxiv.org/abs/2505.08784 · HTML: https://mlsanity.com/paper/2505.08784/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 85 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Conformal Inference of Counterfactuals and Individual Treatment Effects](https://mlsanity.com/paper/2006.06138/) (2020)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Large language model validity via enhanced conformal prediction methods](https://mlsanity.com/paper/2406.09714/) (2024)
- [Exact post-selection inference, with application to the lasso](https://mlsanity.com/paper/1311.6238/) (2013)
- [Confidence Intervals and Hypothesis Testing for High-Dimensional Regression](https://mlsanity.com/paper/1306.3171/) (2013)
- [Optimal Inference After Model Selection](https://mlsanity.com/paper/1410.2597/) (2014)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [OpenML-Python: an extensible Python API for OpenML](https://mlsanity.com/paper/1911.02490/) (2019)
- [Predictive Inference Is Free with the Jackknife+-after-Bootstrap](https://mlsanity.com/paper/2002.09025/) (2020)
- [Exact Post-Selection Inference for Sequential Regression Procedures](https://mlsanity.com/paper/1401.3889/) (2014)
- [Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife](https://mlsanity.com/paper/1311.4555/) (2013)
- [Iterative Random Forests to detect predictive and stable high-order interactions](https://mlsanity.com/paper/1706.08457/) (2017)
- [Merging uncertainty sets via majority vote](https://mlsanity.com/paper/2401.09379/) (2024)
- [Conformal prediction with localization](https://mlsanity.com/paper/1908.08558/) (2019)
- [Stable discovery of interpretable subgroups via calibration in causal studies](https://mlsanity.com/paper/2008.10109/) (2020)
- [KnowGraph: Knowledge-Enabled Anomaly Detection via Logical Reasoning on Graph Data](https://mlsanity.com/paper/2410.08390/) (2024)

## Built on by (most cited first, 9 of 9)

- [CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk](https://mlsanity.com/paper/2507.08150/) (2025)
- [PCS Workflow for Veridical Data Science in the Age of AI](https://mlsanity.com/paper/2508.00835/) (2025)
- [Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection](https://mlsanity.com/paper/2603.27189/) (2026)
- [Cross-World Assumption and Refining Prediction Intervals for Individual Treatment Effects](https://mlsanity.com/paper/2507.12581/) (2025)
- [Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations](https://mlsanity.com/paper/2509.10963/) (2025)
- [Retrospective Counterfactual Prediction by Conditioning on the Factual Outcome: A Cross-World Approach](https://mlsanity.com/paper/2603.27320/) (2026)
- [Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance](https://mlsanity.com/paper/2605.03233/) (2026)
- [Rescuing double robustness: safe estimation under complete misspecification](https://mlsanity.com/paper/2509.22446/) (2025)
- [VDSAgents: A PCS-Guided Multi-Agent System for Veridical Data Science Automation](https://mlsanity.com/paper/2510.24339/) (2025)

## Abstract (arXiv)

As machine learning (ML) enters high-stakes domains, trustworthy uncertainty quantification (UQ) is essential for safety. In this paper we introduce PCS-UQ, a framework based on the Predictability, Computability, and Stability (PCS) principles for veridical data science. Starting with a candidate set of models or algorithms, PCS-UQ integrates a rigorous prediction-check to screen out unsuitable models in the set and utilizes bootstrap samples in order to capture both inter-sample variability and algorithmic instability for the prediction-checked algorithms. We then introduce a novel multiplicative calibration scheme to enhance local adaptivity, which can be viewed as a new score in conformal prediction. Moreover, we produce a compilation of 17 real-world regression datasets with manually constructed subgroups. On this benchmark, PCS-UQ maintains the target coverage while outperforming or matching conformal methods equipped with oracle-selected algorithms in interval width. PCS-UQ achieves consistent subgroup coverage, outperforming these oracle-selected conformal methods. Notably, PCS-UQ stands out in achieving both competitive interval widths and consistent subgroup coverage. Across 6 classification datasets, PCS-UQ reduces prediction set sizes by 20%. To scale the framework for deep learning, we propose computationally efficient variants that bypass expensive retraining. On three computer vision benchmarks, these variants reduce prediction set sizes by 20% over conformal baselines. Finally, we provide a theoretical proof that a modified PCS-UQ algorithm preserves valid coverage under exchangeability as a form of split conformal inference.