# Credal Two-Sample Tests of Epistemic Uncertainty

arXiv: https://arxiv.org/abs/2410.12921 · HTML: https://mlsanity.com/paper/2410.12921/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2024.

## Built on by (most cited first, 16 of 16)

- [Conformalized Credal Regions for Classification with Ambiguous Ground Truth](https://mlsanity.com/paper/2411.04852/) (2024)
- [Integral Imprecise Probability Metrics](https://mlsanity.com/paper/2505.16156/) (2025)
- [Provable Uncertainty Decomposition via Higher-Order Calibration](https://mlsanity.com/paper/2412.18808/) (2024)
- [Credal and Interval Deep Evidential Classifications](https://mlsanity.com/paper/2512.05526/) (2025)
- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Truthful Elicitation of Imprecise Forecasts](https://mlsanity.com/paper/2503.16395/) (2025)
- [Amortized Linear-time Exact Shapley Value for Product-Kernel Methods](https://mlsanity.com/paper/2505.16516/) (2025)
- [Kernel Tests of Equivalence](https://mlsanity.com/paper/2603.10886/) (2026)
- [Quantification of Credal Uncertainty: A Distance-Based Approach](https://mlsanity.com/paper/2603.27270/) (2026)
- [When Do Credal Sets Stabilize? Fixed-Point Theorems for Credal Set Updates](https://mlsanity.com/paper/2510.04769/) (2025)
- [On Uncertainty Calibration for Equivariant Functions](https://mlsanity.com/paper/2510.21691/) (2025)
- [Set-based v.s. Distribution-based Representations of Epistemic Uncertainty: A Comparative Study](https://mlsanity.com/paper/2602.22747/) (2026)
- [Efficient Credal Prediction through Decalibration](https://mlsanity.com/paper/2603.08495/) (2026)
- [Kernel Quantile Embeddings and Associated Probability Metrics](https://mlsanity.com/paper/2505.20433/) (2025)
- [Incentive Aware AI Regulations: A Credal Characterisation](https://mlsanity.com/paper/2603.05175/) (2026)
- [Data Fusion with Distributional Equivalence Test-then-pool](https://mlsanity.com/paper/2603.11867/) (2026)

## Abstract (arXiv)

We introduce credal two-sample testing, a new hypothesis testing framework for comparing credal sets -- convex sets of probability measures where each element captures aleatoric uncertainty and the set itself represents epistemic uncertainty that arises from the modeller's partial ignorance. Compared to classical two-sample tests, which focus on comparing precise distributions, the proposed framework provides a broader and more versatile set of hypotheses. This approach enables the direct integration of epistemic uncertainty, effectively addressing the challenges arising from partial ignorance in hypothesis testing. By generalising two-sample test to compare credal sets, our framework enables reasoning for equality, inclusion, intersection, and mutual exclusivity, each offering unique insights into the modeller's epistemic beliefs. As the first work on nonparametric hypothesis testing for comparing credal sets, we focus on finitely generated credal sets derived from i.i.d. samples from multiple distributions -- referred to as credal samples. We formalise these tests as two-sample tests with nuisance parameters and introduce the first permutation-based solution for this class of problems, significantly improving existing methods. Our approach properly incorporates the modeller's epistemic uncertainty into hypothesis testing, leading to more robust and credible conclusions, with kernel-based implementations for real-world applications.