# Structured Bayesian Pruning via Log-Normal Multiplicative Noise

arXiv: https://arxiv.org/abs/1705.07283 · HTML: https://mlsanity.com/paper/1705.07283/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2017.

## Built on by (most cited first, 30 of 39)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints](https://mlsanity.com/paper/2208.04425/) (2022)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [LANA: Latency Aware Network Acceleration](https://mlsanity.com/paper/2107.10624/) (2021)
- [Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training](https://mlsanity.com/paper/2209.11204/) (2022)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs](https://mlsanity.com/paper/2107.13490/) (2021)
- [Deep Model Compression based on the Training History](https://mlsanity.com/paper/2102.00160/) (2021)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Cyclical Pruning for Sparse Neural Networks](https://mlsanity.com/paper/2202.01290/) (2022)
- [Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence](https://mlsanity.com/paper/2311.15782/) (2023)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty](https://mlsanity.com/paper/2305.00934/) (2023)
- [Can pruning make Large Language Models more efficient?](https://mlsanity.com/paper/2310.04573/) (2023)
- [Pruning in the Face of Adversaries](https://mlsanity.com/paper/2108.08560/) (2021)
- [DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks](https://mlsanity.com/paper/2202.07481/) (2022)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Variational Bayesian Sequence-to-Sequence Networks for Memory-Efficient Sign Language Translation](https://mlsanity.com/paper/2102.06143/) (2021)
- [Speedup deep learning models on GPU by taking advantage of efficient unstructured pruning and bit-width reduction](https://mlsanity.com/paper/2112.15445/) (2021)
- [Quantized Sparse Weight Decomposition for Neural Network Compression](https://mlsanity.com/paper/2207.11048/) (2022)
- [Towards Redundancy-Free Sub-networks in Continual Learning](https://mlsanity.com/paper/2312.00840/) (2023)
- [BMRS: Bayesian Model Reduction for Structured Pruning](https://mlsanity.com/paper/2406.01345/) (2024)
- [Shapley Pruning for Neural Network Compression](https://mlsanity.com/paper/2407.15875/) (2024)
- [A Bayesian Interpretation of Adaptive Low-Rank Adaptation](https://mlsanity.com/paper/2409.10673/) (2024)

## Abstract (arXiv)

Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.