# Efficient Model Compression for Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2411.00273 · HTML: https://mlsanity.com/paper/2411.00273/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 71 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Builds on (most cited first, 23 of 23)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Bayesian Neural Networks: An Introduction and Survey](https://mlsanity.com/paper/2006.12024/) (2020)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Soft Weight-Sharing for Neural Network Compression](https://mlsanity.com/paper/1702.04008/) (2017)
- [The Generalization-Stability Tradeoff In Neural Network Pruning](https://mlsanity.com/paper/1906.03728/) (2019)
- [Model Selection in Bayesian Neural Networks via Horseshoe Priors](https://mlsanity.com/paper/1705.10388/) (2017)
- [Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee](https://mlsanity.com/paper/2011.07439/) (2020)
- [Learning Discrete Weights Using the Local Reparameterization Trick](https://mlsanity.com/paper/1710.07739/) (2017)
- [Consistent Feature Selection for Analytic Deep Neural Networks](https://mlsanity.com/paper/2010.08097/) (2020)
- [Explaining Bayesian Neural Networks](https://mlsanity.com/paper/2108.10346/) (2021)
- [Memory Bounded Deep Convolutional Networks](https://mlsanity.com/paper/1412.1442/) (2014)
- [Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon](https://mlsanity.com/paper/1912.01189/) (2019)
- [Reparameterization trick for discrete variables](https://mlsanity.com/paper/1611.01239/) (2016)
- [Implicit Generative Prior for Bayesian Neural Networks](https://mlsanity.com/paper/2404.18008/) (2024)

## Abstract (arXiv)

Model Compression has drawn much attention within the deep learning community recently. Compressing a dense neural network offers many advantages including lower computation cost, deployability to devices of limited storage and memories, and resistance to adversarial attacks. This may be achieved via weight pruning or fully discarding certain input features. Here we demonstrate a novel strategy to emulate principles of Bayesian model selection in a deep learning setup. Given a fully connected Bayesian neural network with spike-and-slab priors trained via a variational algorithm, we obtain the posterior inclusion probability for every node that typically gets lost. We employ these probabilities for pruning and feature selection on a host of simulated and real-world benchmark data and find evidence of better generalizability of the pruned model in all our experiments.