mlsanityData to 14 Oct 2026

Efficient Model Compression for Bayesian Neural Networks

arXiv 2411.00273 ↗First posted 1 Nov 2024cs.LG stat.AP stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Model Compression has drawn much attention within the deep learning community recently. Compressing a dense neural network offers many advantages including lower computation cost, deployability to devices of limited storage and memories, and resistance to adversarial attacks. This may be achieved via weight pruning or fully discarding certain input features. Here we demonstrate a novel strategy to emulate principles of Bayesian model selection in a deep learning setup. Given a fully connected Bayesian neural network with spike-and-slab priors trained via a variational algorithm, we obtain the posterior inclusion probability for every node that typically gets lost. We employ these probabilities for pruning and feature selection on a host of simulated and real-world benchmark data and find evidence of better generalizability of the pruned model in all our experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 71 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

23 of 71 references.

References · 23

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  4. 2020What is the State of Neural Network Pruning?cited by 583
  5. 2015Weight Uncertainty in Neural Networkscited by 554
  6. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  7. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  8. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  9. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  10. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  11. 2020Bayesian Neural Networks: An Introduction and Surveycited by 131
  12. 2017Bayesian Compression for Deep Learningcited by 116
11 more
  1. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  2. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  3. 2017Model Selection in Bayesian Neural Networks via Horseshoe Priorscited by 52
  4. 2020Efficient Variational Inference for Sparse Deep Learning with Theoretical Guaranteecited by 37
  5. 2017Learning Discrete Weights Using the Local Reparameterization Trickcited by 29
  6. 2020Consistent Feature Selection for Analytic Deep Neural Networkscited by 27
  7. 2021Explaining Bayesian Neural Networkscited by 18
  8. 2014Memory Bounded Deep Convolutional Networkscited by 17
  9. 2019Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenoncited by 4
  10. 2016Reparameterization trick for discrete variablescited by 3
  11. 2024Implicit Generative Prior for Bayesian Neural Networkscited by 1

Built on by

Not cited yet.