# Discriminative calibration: Check Bayesian computation from simulations and flexible classifier

arXiv: https://arxiv.org/abs/2305.14593 · HTML: https://mlsanity.com/paper/2305.14593/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 23 of 23)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Benchmarking Simulation-Based Inference](https://mlsanity.com/paper/2101.04653/) (2021)
- [Validating Bayesian Inference Algorithms with Simulation-Based Calibration](https://mlsanity.com/paper/1804.06788/) (2018)
- [Measuring Sample Quality with Kernels](https://mlsanity.com/paper/1703.01717/) (2017)
- [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019)
- [Real-time gravitational-wave science with neural posterior estimation](https://mlsanity.com/paper/2106.12594/) (2021)
- [Yes, but Did It Work?: Evaluating Variational Inference](https://mlsanity.com/paper/1802.02538/) (2018)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [Sampling-Based Accuracy Testing of Posterior Estimators for General Inference](https://mlsanity.com/paper/2302.03026/) (2023)
- [Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization](https://mlsanity.com/paper/2006.10343/) (2020)
- [GATSBI: Generative Adversarial Training for Simulation-Based Inference](https://mlsanity.com/paper/2203.06481/) (2022)
- [A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful](https://mlsanity.com/paper/2110.06581/) (2021)
- [Visualization in Bayesian workflow](https://mlsanity.com/paper/1709.01449/) (2017)
- [Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference](https://mlsanity.com/paper/2011.05836/) (2020)
- [Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity](https://mlsanity.com/paper/2211.02383/) (2022)
- [A Deep Learning Method for Comparing Bayesian Hierarchical Models](https://mlsanity.com/paper/2301.11873/) (2023)
- [Diagnostic tools of approximate Bayesian computation using the coverage property](https://mlsanity.com/paper/1301.3166/) (2013)
- [Validation Diagnostics for SBI algorithms based on Normalizing Flows](https://mlsanity.com/paper/2211.09602/) (2022)
- [An Easy to Interpret Diagnostic for Approximate Inference: Symmetric Divergence Over Simulations](https://mlsanity.com/paper/2103.01030/) (2021)

## Built on by (most cited first, 2 of 2)

- [L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inference](https://mlsanity.com/paper/2306.03580/) (2023)
- [Simulation-based stacking](https://mlsanity.com/paper/2310.17009/) (2023)

## Abstract (arXiv)

To check the accuracy of Bayesian computations, it is common to use rank-based simulation-based calibration (SBC). However, SBC has drawbacks: The test statistic is somewhat ad-hoc, interactions are difficult to examine, multiple testing is a challenge, and the resulting p-value is not a divergence metric. We propose to replace the marginal rank test with a flexible classification approach that learns test statistics from data. This measure typically has a higher statistical power than the SBC rank test and returns an interpretable divergence measure of miscalibration, computed from classification accuracy. This approach can be used with different data generating processes to address likelihood-free inference or traditional inference methods like Markov chain Monte Carlo or variational inference. We illustrate an automated implementation using neural networks and statistically-inspired features, and validate the method with numerical and real data experiments.