# Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors

arXiv: https://arxiv.org/abs/1806.05975 · HTML: https://mlsanity.com/paper/1806.05975/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2018.

## Built on by (most cited first, 30 of 35)

- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data](https://mlsanity.com/paper/2210.13043/) (2022)
- [Understanding Event-Generation Networks via Uncertainties](https://mlsanity.com/paper/2104.04543/) (2021)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [Data-SUITE: Data-centric identification of in-distribution incongruous examples](https://mlsanity.com/paper/2202.08836/) (2022)
- [Flat Seeking Bayesian Neural Networks](https://mlsanity.com/paper/2302.02713/) (2023)
- [On the detrimental effect of invariances in the likelihood for variational inference](https://mlsanity.com/paper/2209.07157/) (2022)
- [Are you using test log-likelihood correctly?](https://mlsanity.com/paper/2212.00219/) (2022)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks](https://mlsanity.com/paper/2408.05496/) (2024)
- [Fast Conditional Network Compression Using Bayesian HyperNetworks](https://mlsanity.com/paper/2205.06404/) (2022)
- [What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization](https://mlsanity.com/paper/2207.05161/) (2022)
- [Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty](https://mlsanity.com/paper/2305.00934/) (2023)
- [Improving Generalization with Flat Hilbert Bayesian Inference](https://mlsanity.com/paper/2410.04196/) (2024)
- [Merging Two Cultures: Deep and Statistical Learning](https://mlsanity.com/paper/2110.11561/) (2021)
- [Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks](https://mlsanity.com/paper/2307.07753/) (2023)
- [Class Incremental Online Streaming Learning](https://mlsanity.com/paper/2110.10741/) (2021)
- [Encoding Domain Knowledge in Multi-view Latent Variable Models: A Bayesian Approach with Structured Sparsity](https://mlsanity.com/paper/2204.06242/) (2022)
- [Streaming LifeLong Learning With Any-Time Inference](https://mlsanity.com/paper/2301.11892/) (2023)
- [Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models](https://mlsanity.com/paper/2506.07247/) (2025)
- [Revisiting Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2103.12496/) (2021)
- [Are Visual Recognition Models Robust to Image Compression?](https://mlsanity.com/paper/2304.04518/) (2023)
- [Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning](https://mlsanity.com/paper/2407.02721/) (2024)
- [Incorporating functional summary information in Bayesian neural networks using a Dirichlet process likelihood approach](https://mlsanity.com/paper/2207.01234/) (2022)
- [Bayesian sparsification for deep neural networks with Bayesian model reduction](https://mlsanity.com/paper/2309.12095/) (2023)
- [DAGnosis: Localized Identification of Data Inconsistencies using Structures](https://mlsanity.com/paper/2402.17599/) (2024)
- [Vector Copula Variational Inference and Dependent Block Posterior Approximations](https://mlsanity.com/paper/2503.01072/) (2025)
- [Stochastic Collapsed Variational Inference for Structured Gaussian Process Regression Network](https://mlsanity.com/paper/2106.00719/) (2021)

## Abstract (arXiv)

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. Recent work has proposed the use of a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. In this work, we propose several modeling and inference advances that consistently improve the compactness of the model learned while maintaining predictive performance, especially in smaller-sample settings including reinforcement learning.