# Conformal prediction under ambiguous ground truth

arXiv: https://arxiv.org/abs/2307.09302 · HTML: https://mlsanity.com/paper/2307.09302/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [Learning Optimal Conformal Classifiers](https://mlsanity.com/paper/2110.09192/) (2021)
- [Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions](https://mlsanity.com/paper/2104.03829/) (2021)
- [Predictive Inference Is Free with the Jackknife+-after-Bootstrap](https://mlsanity.com/paper/2002.09025/) (2020)
- [Conformal Prediction Sets with Limited False Positives](https://mlsanity.com/paper/2202.07650/) (2022)
- [Minimax-optimal Inference from Partial Rankings](https://mlsanity.com/paper/1406.5638/) (2014)
- [Choosing Between Methods of Combining p-values](https://mlsanity.com/paper/1707.06897/) (2017)
- [Discretized conformal prediction for efficient distribution-free inference](https://mlsanity.com/paper/1709.06233/) (2017)
- [A Cross-Conformal Predictor for Multi-label Classification](https://mlsanity.com/paper/2211.16238/) (2022)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [Partition-Mallows Model and Its Inference for Rank Aggregation](https://mlsanity.com/paper/2104.07261/) (2021)

## Built on by (most cited first, 30 of 31)

- [SynthID-Image: Image watermarking at internet scale](https://mlsanity.com/paper/2510.09263/) (2025)
- [E-Values Expand the Scope of Conformal Prediction](https://mlsanity.com/paper/2503.13050/) (2025)
- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [Conformalized Credal Set Predictors](https://mlsanity.com/paper/2402.10723/) (2024)
- [Conformalized Credal Regions for Classification with Ambiguous Ground Truth](https://mlsanity.com/paper/2411.04852/) (2024)
- [Weighted Aggregation of Conformity Scores for Classification](https://mlsanity.com/paper/2407.10230/) (2024)
- [Integral Imprecise Probability Metrics](https://mlsanity.com/paper/2505.16156/) (2025)
- [Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data](https://mlsanity.com/paper/2310.09926/) (2023)
- [Synthetic-Powered Predictive Inference](https://mlsanity.com/paper/2505.13432/) (2025)
- [Entropy Reweighted Conformal Classification](https://mlsanity.com/paper/2407.17377/) (2024)
- [Improving the statistical efficiency of cross-conformal prediction](https://mlsanity.com/paper/2503.01495/) (2025)
- [Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data](https://mlsanity.com/paper/2507.10425/) (2025)
- [Adaptive Conformal Inference by Particle Filtering under Hidden Markov Models](https://mlsanity.com/paper/2411.01558/) (2024)
- [Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry](https://mlsanity.com/paper/2602.01667/) (2026)
- [Reliable Multi-View Learning with Conformal Prediction for Aortic Stenosis Classification in Echocardiography](https://mlsanity.com/paper/2409.09680/) (2024)
- [Distilling Calibration via Conformalized Credal Inference](https://mlsanity.com/paper/2501.06066/) (2025)
- [Conformal Prediction with Corrupted Labels: Uncertain Imputation and Robust Re-weighting](https://mlsanity.com/paper/2505.04733/) (2025)
- [Optimal Conformal Prediction under Epistemic Uncertainty](https://mlsanity.com/paper/2505.19033/) (2025)
- [Conformal Prediction for Signal Temporal Logic Inference](https://mlsanity.com/paper/2509.25473/) (2025)
- [Meta-Analysis with Untrusted Data](https://mlsanity.com/paper/2407.09387/) (2024)
- [Confidence Calibration under Ambiguous Ground Truth](https://mlsanity.com/paper/2603.22879/) (2026)
- [Elements of Conformal Prediction](https://mlsanity.com/paper/2603.23923/) (2026)
- [Concise and Logically Consistent Conformal Sets for Neuro-Symbolic Concept-Based Models](https://mlsanity.com/paper/2605.18202/) (2026)
- [Empirical Bayes Conformal Prediction for Vision and Language Models](https://mlsanity.com/paper/2605.23189/) (2026)
- [Uncertainty-Aware Generation and Decision-Making Under Ambiguity](https://mlsanity.com/paper/2606.30578/) (2026)
- [Optimal Transport for ε-Contaminated Credal Sets: To the Memory of Sayan Mukherjee](https://mlsanity.com/paper/2410.03267/) (2024)
- [Improving Coverage in Combined Prediction Sets with Weighted p-values](https://mlsanity.com/paper/2505.11785/) (2025)
- [CertDW: Towards Certified Dataset Ownership Verification via Conformal Calibration](https://mlsanity.com/paper/2506.13160/) (2025)
- [Clinical Uncertainty Impacts Machine Learning Evaluations](https://mlsanity.com/paper/2509.22242/) (2025)
- [Neural Conjugate Aggregation: Identifiable Unsupervised Multi-Sensor Regression under Heterogeneous Sensor Bias](https://mlsanity.com/paper/2606.22200/) (2026)

## Abstract (arXiv)

Conformal Prediction (CP) allows to perform rigorous uncertainty quantification by constructing a prediction set C(X) satisfying mathbbP(Y ∈ C(X))≥ 1-α for a user-chosen α ∈ [0,1] by relying on calibration data (X1,Y1),...,(Xn,Yn) from mathbbP=mathbbPX otimes mathbbPY|X. It is typically implicitly assumed that mathbbPY|X is the "true" posterior label distribution. However, in many real-world scenarios, the labels Y1,...,Yn are obtained by aggregating expert opinions using a voting procedure, resulting in a one-hot distribution mathbbPvoteY|X. For such ``voted'' labels, CP guarantees are thus w.r.t. mathbbPvote=mathbbPX otimes mathbbPvoteY|X rather than the true distribution mathbbP. In cases with unambiguous ground truth labels, the distinction between mathbbPvote and mathbbP is irrelevant. However, when experts do not agree because of ambiguous labels, approximating mathbbPY|X with a one-hot distribution mathbbPvoteY|X ignores this uncertainty. In this paper, we propose to leverage expert opinions to approximate mathbbPY|X using a non-degenerate distribution mathbbPaggY|X. We develop Monte Carlo CP procedures which provide guarantees w.r.t. mathbbPagg=mathbbPX otimes mathbbPaggY|X by sampling multiple synthetic pseudo-labels from mathbbPaggY|X for each calibration example X1,...,Xn. In a case study of skin condition classification with significant disagreement among expert annotators, we show that applying CP w.r.t. mathbbPvote under-covers expert annotations: calibrated for 72% coverage, it falls short by on average 10%; our Monte Carlo CP closes this gap both empirically and theoretically.