# Provable Uncertainty Decomposition via Higher-Order Calibration

arXiv: https://arxiv.org/abs/2412.18808 · HTML: https://mlsanity.com/paper/2412.18808/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 25 Dec 2024.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Conformal Prediction With Conditional Guarantees](https://mlsanity.com/paper/2305.12616/) (2023)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation](https://mlsanity.com/paper/2203.06102/) (2022)
- [Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?](https://mlsanity.com/paper/2306.09586/) (2023)
- [Uncertainty Estimation Using a Single Deep Deterministic Neural Network](https://mlsanity.com/paper/2003.02037/) (2020)
- [Second-Order Uncertainty Quantification: A Distance-Based Approach](https://mlsanity.com/paper/2312.00995/) (2023)
- [Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?](https://mlsanity.com/paper/2402.09056/) (2024)
- [From Predictions to Decisions: The Importance of Joint Predictive Distributions](https://mlsanity.com/paper/2107.09224/) (2021)
- [Learn to Accumulate Evidence from All Training Samples: Theory and Practice](https://mlsanity.com/paper/2306.11113/) (2023)
- [Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' View](https://mlsanity.com/paper/2305.16703/) (2023)
- [Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty](https://mlsanity.com/paper/2311.08309/) (2023)
- [From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation](https://mlsanity.com/paper/2402.10727/) (2024)
- [Uncertainty Estimation in Autoregressive Structured Prediction](https://mlsanity.com/paper/2002.07650/) (2020)
- [Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules](https://mlsanity.com/paper/2404.12215/) (2024)
- [Second-Order Uncertainty Quantification: Variance-Based Measures](https://mlsanity.com/paper/2401.00276/) (2023)
- [Conformalized Credal Regions for Classification with Ambiguous Ground Truth](https://mlsanity.com/paper/2411.04852/) (2024)
- [On Computationally Efficient Multi-Class Calibration](https://mlsanity.com/paper/2402.07821/) (2024)
- [Credal Two-Sample Tests of Epistemic Uncertainty](https://mlsanity.com/paper/2410.12921/) (2024)
- [ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference](https://mlsanity.com/paper/2211.11435/) (2022)
- [Distribution-free inference with hierarchical data](https://mlsanity.com/paper/2306.06342/) (2023)

## Built on by (most cited first, 9 of 9)

- [The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguity](https://mlsanity.com/paper/2511.04418/) (2025)
- [When does a predictor know its own loss?](https://mlsanity.com/paper/2502.20375/) (2025)
- [Multidimensional Uncertainty Quantification via Optimal Transport](https://mlsanity.com/paper/2509.22380/) (2025)
- [Auditability and the Landscape of Distance to Multicalibration](https://mlsanity.com/paper/2509.16930/) (2025)
- [Pareto-Guided Optimization for Uncertainty-Aware Medical Image Segmentation](https://mlsanity.com/paper/2601.19365/) (2026)
- [Flexible Routing via Uncertainty Decomposition](https://mlsanity.com/paper/2605.07805/) (2026)
- [The Minimax Rate of Perturbed Second-Order Calibration](https://mlsanity.com/paper/2605.07808/) (2026)
- [Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model](https://mlsanity.com/paper/2610.01006/) (2026)
- [ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty](https://mlsanity.com/paper/2610.17016/) (2026)

## Abstract (arXiv)

We give a principled method for decomposing the predictive uncertainty of a model into aleatoric and epistemic components with explicit semantics relating them to the real-world data distribution. While many works in the literature have proposed such decompositions, they lack the type of formal guarantees we provide. Our method is based on the new notion of higher-order calibration, which generalizes ordinary calibration to the setting of higher-order predictors that predict mixtures over label distributions at every point. We show how to measure as well as achieve higher-order calibration using access to k-snapshots, namely examples where each point has k independent conditional labels. Under higher-order calibration, the estimated aleatoric uncertainty at a point is guaranteed to match the real-world aleatoric uncertainty averaged over all points where the prediction is made. To our knowledge, this is the first formal guarantee of this type that places no assumptions whatsoever on the real-world data distribution. Importantly, higher-order calibration is also applicable to existing higher-order predictors such as Bayesian and ensemble models and provides a natural evaluation metric for such models. We demonstrate through experiments that our method produces meaningful uncertainty decompositions for image classification.