# Meta-Uncertainty in Bayesian Model Comparison

arXiv: https://arxiv.org/abs/2210.07278 · HTML: https://mlsanity.com/paper/2210.07278/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 89 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [BayesFlow: Learning complex stochastic models with invertible neural networks](https://mlsanity.com/paper/2003.06281/) (2020)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [Concentration of tempered posteriors and of their variational approximations](https://mlsanity.com/paper/1706.09293/) (2017)
- [Generalized Variational Inference: Three arguments for deriving new Posteriors](https://mlsanity.com/paper/1904.02063/) (2019)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks](https://mlsanity.com/paper/2112.08866/) (2021)
- [Convergence Rates of Variational Posterior Distributions](https://mlsanity.com/paper/1712.02519/) (2017)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [On the marginal likelihood and cross-validation](https://mlsanity.com/paper/1905.08737/) (2019)
- [Amortized Bayesian model comparison with evidential deep learning](https://mlsanity.com/paper/2004.10629/) (2020)
- [Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy](https://mlsanity.com/paper/2209.02439/) (2022)
- [Robust Approximate Bayesian Inference with Synthetic Likelihood](https://mlsanity.com/paper/1904.04551/) (2019)
- [Objective Bayesian inference with proper scoring rules](https://mlsanity.com/paper/1711.10819/) (2017)
- [Generalized Evidence Theory](https://mlsanity.com/paper/1404.4801/) (2014)
- [Approximate leave-future-out cross-validation for Bayesian time series models](https://mlsanity.com/paper/1902.06281/) (2019)
- [Projective Inference in High-dimensional Problems: Prediction and Feature Selection](https://mlsanity.com/paper/1810.02406/) (2018)
- [Diagnosing model misspecification and performing generalized Bayes' updates via probabilistic classifiers](https://mlsanity.com/paper/1912.05810/) (2019)
- [Workflow Techniques for the Robust Use of Bayes Factors](https://mlsanity.com/paper/2103.08744/) (2021)
- [Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior](https://mlsanity.com/paper/2208.07132/) (2022)

## Built on by (most cited first, 5 of 5)

- [BayesFlow: Amortized Bayesian Workflows With Neural Networks](https://mlsanity.com/paper/2306.16015/) (2023)
- [A Deep Learning Method for Comparing Bayesian Hierarchical Models](https://mlsanity.com/paper/2301.11873/) (2023)
- [Misspecification-robust Sequential Neural Likelihood for Simulation-based Inference](https://mlsanity.com/paper/2301.13368/) (2023)
- [Neural Superstatistics for Bayesian Estimation of Dynamic Cognitive Models](https://mlsanity.com/paper/2211.13165/) (2022)
- [Automating Model Comparison in Factor Graphs](https://mlsanity.com/paper/2306.05965/) (2023)

## Abstract (arXiv)

Bayesian model comparison (BMC) offers a principled probabilistic approach to study and rank competing models. In standard BMC, we construct a discrete probability distribution over the set of possible models, conditional on the observed data of interest. These posterior model probabilities (PMPs) are measures of uncertainty, but -- when derived from a finite number of observations -- are also uncertain themselves. In this paper, we conceptualize distinct levels of uncertainty which arise in BMC. We explore a fully probabilistic framework for quantifying meta-uncertainty, resulting in an applied method to enhance any BMC workflow. Drawing on both Bayesian and frequentist techniques, we represent the uncertainty over the uncertain PMPs via meta-models which combine simulated and observed data into a predictive distribution for PMPs on new data. We demonstrate the utility of the proposed method in the context of conjugate Bayesian regression, likelihood-based inference with Markov chain Monte Carlo, and simulation-based inference with neural networks.