# A Bayesian information criterion for singular models

arXiv: https://arxiv.org/abs/1309.0911 · HTML: https://mlsanity.com/paper/1309.0911/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2013.

## Built on by (most cited first, 22 of 22)

- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [Asymptotic Behavior of Bayesian Generalization Error in Multinomial Mixtures](https://mlsanity.com/paper/2203.06884/) (2022)
- [Mathematical Theory of Bayesian Statistics for Unknown Information Source](https://mlsanity.com/paper/2206.05630/) (2022)
- [Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Cases](https://mlsanity.com/paper/2303.15739/) (2023)
- [Recent Advances in Algebraic Geometry and Bayesian Statistics](https://mlsanity.com/paper/2211.10049/) (2022)
- [Bayesian Model Selection via Mean-Field Variational Approximation](https://mlsanity.com/paper/2312.10607/) (2023)
- [Choosing the Number of Topics in LDA Models -- A Monte Carlo Comparison of Selection Criteria](https://mlsanity.com/paper/2212.14074/) (2022)
- [Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation](https://mlsanity.com/paper/2303.09154/) (2023)
- [Uncertainty-Aware Diagnostics for Physics-Informed Machine Learning](https://mlsanity.com/paper/2510.26121/) (2025)
- [Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theory](https://mlsanity.com/paper/2406.10234/) (2024)
- [Bayesian nonparametric mixture inconsistency for the number of components: How worried should we be in practice?](https://mlsanity.com/paper/2207.14717/) (2022)
- [Learning under Singularity: An Information Criterion improving WBIC and sBIC](https://mlsanity.com/paper/2402.12762/) (2024)
- [Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBM](https://mlsanity.com/paper/2403.09206/) (2024)
- [Debiased maximum-likelihood estimators for hazard ratios under kernel-based machine-learning adjustment](https://mlsanity.com/paper/2507.17686/) (2025)
- [Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents](https://mlsanity.com/paper/2603.03401/) (2026)
- [Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks](https://mlsanity.com/paper/2603.12785/) (2026)
- [The Parametric Stability of Well-separated Spherical Gaussian Mixtures](https://mlsanity.com/paper/2302.00242/) (2023)
- [Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Network](https://mlsanity.com/paper/2508.17783/) (2025)
- [Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models](https://mlsanity.com/paper/2511.06407/) (2025)
- [PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory](https://mlsanity.com/paper/2604.17219/) (2026)
- [Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models](https://mlsanity.com/paper/2608.20183/) (2026)
- [Singularities of Non-negative Matrix Factorization and their application to Bayesian inference](https://mlsanity.com/paper/2609.34043/) (2026)

## Abstract (arXiv)

We consider approximate Bayesian model choice for model selection problems that involve models whose Fisher-information matrices may fail to be invertible along other competing submodels. Such singular models do not obey the regularity conditions underlying the derivation of Schwarz's Bayesian information criterion (BIC) and the penalty structure in BIC generally does not reflect the frequentist large-sample behavior of their marginal likelihood. While large-sample theory for the marginal likelihood of singular models has been developed recently, the resulting approximations depend on the true parameter value and lead to a paradox of circular reasoning. Guided by examples such as determining the number of components of mixture models, the number of factors in latent factor models or the rank in reduced-rank regression, we propose a resolution to this paradox and give a practical extension of BIC for singular model selection problems.