# Singular Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2602.00387 · HTML: https://mlsanity.com/paper/2602.00387/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2026.

## Built on by (most cited first, 4 of 4)

- [Stabilizing Native Low-Rank LLM Pretraining](https://mlsanity.com/paper/2602.12429/) (2026)
- [Not Just How Much, But Where: Decomposing Epistemic Uncertainty into Per-Class Contributions](https://mlsanity.com/paper/2602.21160/) (2026)
- [Perturbation-Based Epistemic Uncertainty for Failure Detection in Vision-Language-Action Models](https://mlsanity.com/paper/2606.20754/) (2026)
- [Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancing](https://mlsanity.com/paper/2609.32271/) (2026)

## Abstract (arXiv)

Bayesian neural networks promise calibrated uncertainty but require O(mn) parameters for standard mean-field Gaussian posteriors. We argue this cost is often unnecessary, particularly when weight matrices exhibit fast singular value decay. By parameterizing weights as W = ABtop with A ∈ ℝm × r, B ∈ ℝn × r, we induce a posterior that is singular with respect to the Lebesgue measure, concentrating on the rank-r manifold. This singularity captures structured weight correlations through shared latent factors, geometrically distinct from mean-field's independence assumption. We derive PAC-Bayes generalization bounds whose complexity term scales as √r(m+n) instead of √m n, and prove loss bounds that decompose the error into optimization and rank-induced bias using the Eckart-Young-Mirsky theorem. We further adapt recent Gaussian complexity bounds for low-rank deterministic networks to Bayesian predictive means. Empirically, across MLPs, LSTMs, and Transformers on standard benchmarks, our method achieves competitive predictive performance while using up to 33× fewer parameters than 5-member Deep Ensembles. It substantially improves OOD detection and often improves calibration relative to mean-field and perturbation baselines, while Deep Ensembles can still be stronger on in-distribution likelihood-based metrics.