# Conformalized Credal Regions for Classification with Ambiguous Ground Truth

arXiv: https://arxiv.org/abs/2411.04852 · HTML: https://mlsanity.com/paper/2411.04852/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 72 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation](https://mlsanity.com/paper/2203.06102/) (2022)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Credal Bayesian Deep Learning](https://mlsanity.com/paper/2302.09656/) (2023)
- [Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?](https://mlsanity.com/paper/2306.09586/) (2023)
- [Optimal Testing for Properties of Distributions](https://mlsanity.com/paper/1507.05952/) (2015)
- [Second-Order Uncertainty Quantification: A Distance-Based Approach](https://mlsanity.com/paper/2312.00995/) (2023)
- [Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?](https://mlsanity.com/paper/2402.09056/) (2024)
- [Conformal prediction under ambiguous ground truth](https://mlsanity.com/paper/2307.09302/) (2023)
- [Credal Learning Theory](https://mlsanity.com/paper/2402.00957/) (2024)
- [Learn to Accumulate Evidence from All Training Samples: Theory and Practice](https://mlsanity.com/paper/2306.11113/) (2023)
- [Robust Hypothesis Testing Using Wasserstein Uncertainty Sets](https://mlsanity.com/paper/1805.10611/) (2018)
- [Conformalized Credal Set Predictors](https://mlsanity.com/paper/2402.10723/) (2024)
- [Credal Two-Sample Tests of Epistemic Uncertainty](https://mlsanity.com/paper/2410.12921/) (2024)
- [A Novel Bayes' Theorem for Upper Probabilities](https://mlsanity.com/paper/2307.06831/) (2023)
- [Distributionally Robust Statistical Verification with Imprecise Neural Networks](https://mlsanity.com/paper/2308.14815/) (2023)
- [Validity, consonant plausibility measures, and conformal prediction](https://mlsanity.com/paper/2001.09225/) (2020)
- [On the Calibration of Probabilistic Classifier Sets](https://mlsanity.com/paper/2205.10082/) (2022)
- [Conformal Credal Self-Supervised Learning](https://mlsanity.com/paper/2205.15239/) (2022)
- [Valid inferential models for prediction in supervised learning problems](https://mlsanity.com/paper/2112.10234/) (2021)
- [DC4L: Distribution Shift Recovery via Data-Driven Control for Deep Learning Models](https://mlsanity.com/paper/2302.10341/) (2023)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)

## Built on by (most cited first, 19 of 19)

- [Integral Imprecise Probability Metrics](https://mlsanity.com/paper/2505.16156/) (2025)
- [Provable Uncertainty Decomposition via Higher-Order Calibration](https://mlsanity.com/paper/2412.18808/) (2024)
- [Conformal Prediction Regions are Imprecise Highest Density Regions](https://mlsanity.com/paper/2502.06331/) (2025)
- [Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data](https://mlsanity.com/paper/2507.10425/) (2025)
- [Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry](https://mlsanity.com/paper/2602.01667/) (2026)
- [Distilling Calibration via Conformalized Credal Inference](https://mlsanity.com/paper/2501.06066/) (2025)
- [Conformal Prediction with Corrupted Labels: Uncertain Imputation and Robust Re-weighting](https://mlsanity.com/paper/2505.04733/) (2025)
- [Optimal Conformal Prediction under Epistemic Uncertainty](https://mlsanity.com/paper/2505.19033/) (2025)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [Bayesian Conformal Prediction as a Decision Risk Problem](https://mlsanity.com/paper/2602.03331/) (2026)
- [When Do Credal Sets Stabilize? Fixed-Point Theorems for Credal Set Updates](https://mlsanity.com/paper/2510.04769/) (2025)
- [Concise and Logically Consistent Conformal Sets for Neuro-Symbolic Concept-Based Models](https://mlsanity.com/paper/2605.18202/) (2026)
- [Empirical Bayes Conformal Prediction for Vision and Language Models](https://mlsanity.com/paper/2605.23189/) (2026)
- [Are Hallucinations Bad Estimations?](https://mlsanity.com/paper/2509.21473/) (2025)
- [Adaptive Conformal Prediction for Quantum Machine Learning](https://mlsanity.com/paper/2511.18225/) (2025)
- [CREDO: Epistemic-Aware Conformalized Credal Envelopes for Regression](https://mlsanity.com/paper/2603.06826/) (2026)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)
- [Decoupled Conformal Optimisation: Efficient Prediction Sets via Independent Tuning and Calibration](https://mlsanity.com/paper/2605.18354/) (2026)
- [On the QUEST for Uncertainty Quantification via Highest Density Regions](https://mlsanity.com/paper/2606.19569/) (2026)

## Abstract (arXiv)

An open question in Imprecise Probabilistic Machine Learning is how to empirically derive a credal region (i.e., a closed and convex family of probabilities on the output space) from the available data, without any prior knowledge or assumption. In classification problems, credal regions are a tool that is able to provide provable guarantees under realistic assumptions by characterizing the uncertainty about the distribution of the labels. Building on previous work, we show that credal regions can be directly constructed using conformal methods. This allows us to provide a novel extension of classical conformal prediction to problems with ambiguous ground truth, that is, when the exact labels for given inputs are not exactly known. The resulting construction enjoys desirable practical and theoretical properties: (i) conformal coverage guarantees, (ii) smaller prediction sets (compared to classical conformal prediction regions) and (iii) disentanglement of uncertainty sources (epistemic, aleatoric). We empirically verify our findings on both synthetic and real datasets.