# A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI

arXiv: https://arxiv.org/abs/2609.24419 · HTML: https://mlsanity.com/paper/2609.24419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows](https://mlsanity.com/paper/1805.07226/) (2018)
- [Automatic Posterior Transformation for Likelihood-Free Inference](https://mlsanity.com/paper/1905.07488/) (2019)
- [Flexible statistical inference for mechanistic models of neural dynamics](https://mlsanity.com/paper/1711.01861/) (2017)
- [On Contrastive Learning for Likelihood-free Inference](https://mlsanity.com/paper/2002.03712/) (2020)
- [Approximating Likelihood Ratios with Calibrated Discriminative Classifiers](https://mlsanity.com/paper/1506.02169/) (2015)
- [Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models](https://mlsanity.com/paper/1501.03291/) (2015)
- [Deep Learning in Bioinformatics](https://mlsanity.com/paper/1603.06430/) (2016)
- [Compositional Score Modeling for Simulation-based Inference](https://mlsanity.com/paper/2209.14249/) (2022)
- [Truncated proposals for scalable and hassle-free simulation-based inference](https://mlsanity.com/paper/2210.04815/) (2022)
- [Simulation-Based Inference: A Practical Guide](https://mlsanity.com/paper/2508.12939/) (2025)
- [High-Dimensional Density Ratio Estimation with Extensions to Approximate Likelihood Computation](https://mlsanity.com/paper/1404.7063/) (2014)
- [HNPE: Leveraging Global Parameters for Neural Posterior Estimation](https://mlsanity.com/paper/2102.06477/) (2021)
- [Diffusion posterior sampling for simulation-based inference in tall data settings](https://mlsanity.com/paper/2404.07593/) (2024)
- [Confidence Sets and Hypothesis Testing in a Likelihood-Free Inference Setting](https://mlsanity.com/paper/2002.10399/) (2020)
- [Simulator-Based Inference with Waldo: Confidence Regions by Leveraging Prediction Algorithms and Posterior Estimators for Inverse Problems](https://mlsanity.com/paper/2205.15680/) (2022)
- [Regression Trees for Fast and Adaptive Prediction Intervals](https://mlsanity.com/paper/2402.07357/) (2024)
- [Likelihood-Free Frequentist Inference: Bridging Classical Statistics and Machine Learning for Reliable Simulator-Based Inference](https://mlsanity.com/paper/2107.03920/) (2021)
- [Epistemic Uncertainty in Conformal Scores: A Unified Approach](https://mlsanity.com/paper/2502.06995/) (2025)
- [ABC-CDE: Towards Approximate Bayesian Computation with Complex High-Dimensional Data and Limited Simulations](https://mlsanity.com/paper/1805.05480/) (2018)
- [Approximate Bayesian Computation with Deep Learning and Conformal prediction](https://mlsanity.com/paper/2406.04874/) (2024)
- [CP4SBI: Local Conformal Calibration of Credible Sets in Simulation-Based Inference](https://mlsanity.com/paper/2508.17077/) (2025)
- [REACT to NHST: Sensible conclusions to meaningful hypotheses](https://mlsanity.com/paper/2308.09112/) (2023)
- [Simulation-based Inference of Developmental EEG Maturation with the Spectral Graph Model](https://mlsanity.com/paper/2405.02524/) (2024)

## Abstract (arXiv)

Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods. A primary goal in these settings is to obtain credible regions with valid coverage. While recent model-agnostic conformal calibration methods have succeeded in constructing credible sets with prescribed local Bayesian coverage, their approximate nature introduces inherent epistemic uncertainty in the calibration process. In this work, we propose a novel tool for diagnosing calibration uncertainty. Our approach is based on a simple three-way hypothesis testing procedure. We demonstrate how this tool can be used to effectively assess necessary simulation budgets for calibration sets and analyze the epistemic uncertainty associated with cutoff estimation.