# Mathematical Theory of Bayesian Statistics for Unknown Information Source

arXiv: https://arxiv.org/abs/2206.05630 · HTML: https://mlsanity.com/paper/2206.05630/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 47 references that are in the map.
- First posted on arXiv 11 Jun 2022.

## Builds on (most cited first, 7 of 7)

- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [A Bayesian information criterion for singular models](https://mlsanity.com/paper/1309.0911/) (2013)
- [The Exact Asymptotic Form of Bayesian Generalization Error in Latent Dirichlet Allocation](https://mlsanity.com/paper/2008.01304/) (2020)
- [Approximate leave-future-out cross-validation for Bayesian time series models](https://mlsanity.com/paper/1902.06281/) (2019)
- [Asymptotic Behavior of Bayesian Generalization Error in Multinomial Mixtures](https://mlsanity.com/paper/2203.06884/) (2022)

## Built on by (most cited first, 5 of 5)

- [Recent Advances in Algebraic Geometry and Bayesian Statistics](https://mlsanity.com/paper/2211.10049/) (2022)
- [Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation](https://mlsanity.com/paper/2303.09154/) (2023)
- [Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theory](https://mlsanity.com/paper/2406.10234/) (2024)
- [Learning under Singularity: An Information Criterion improving WBIC and sBIC](https://mlsanity.com/paper/2402.12762/) (2024)
- [Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBM](https://mlsanity.com/paper/2403.09206/) (2024)

## Abstract (arXiv)

In statistical inference, uncertainty is unknown and all models are wrong. That is to say, a person who makes a statistical model and a prior distribution is simultaneously aware that both are fictional candidates. To study such cases, statistical measures have been constructed, such as cross validation, information criteria, and marginal likelihood, however, their mathematical properties have not yet been completely clarified when statistical models are under- and over- parametrized. We introduce a place of mathematical theory of Bayesian statistics for unknown uncertainty, which clarifies general properties of cross validation, information criteria, and marginal likelihood, even if an unknown data-generating process is unrealizable by a model or even if the posterior distribution cannot be approximated by any normal distribution. Hence it gives a helpful standpoint for a person who cannot believe in any specific model and prior. This paper consists of three parts. The first is a new result, whereas the second and third are well-known previous results with new experiments. We show there exists a more precise estimator of the generalization loss than leave-one-out cross validation, there exists a more accurate approximation of marginal likelihood than BIC, and the optimal hyperparameters for generalization loss and marginal likelihood are different.