# BMRS: Bayesian Model Reduction for Structured Pruning

arXiv: https://arxiv.org/abs/2406.01345 · HTML: https://mlsanity.com/paper/2406.01345/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models](https://mlsanity.com/paper/2007.03051/) (2020)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Model Selection in Bayesian Neural Networks via Horseshoe Priors](https://mlsanity.com/paper/1705.10388/) (2017)
- [Structured Bayesian Pruning via Log-Normal Multiplicative Noise](https://mlsanity.com/paper/1705.07283/) (2017)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee](https://mlsanity.com/paper/2011.07439/) (2020)
- [Constructing Fast Network through Deconstruction of Convolution](https://mlsanity.com/paper/1806.07370/) (2018)
- [Bayesian model reduction](https://mlsanity.com/paper/1805.07092/) (2018)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AI](https://mlsanity.com/paper/2309.02065/) (2023)
- [Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation](https://mlsanity.com/paper/2307.09701/) (2023)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization](https://mlsanity.com/paper/2210.09134/) (2022)
- [Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty](https://mlsanity.com/paper/2305.00934/) (2023)
- [Bayesian sparsification for deep neural networks with Bayesian model reduction](https://mlsanity.com/paper/2309.12095/) (2023)

## Built on by (most cited first, 2 of 2)

- [Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery](https://mlsanity.com/paper/2411.09127/) (2024)
- [Rethinking Reservoir Pruning: A Dynamical Perspective for Echo State Networks](https://mlsanity.com/paper/2608.04593/) (2026)

## Abstract (arXiv)

Modern neural networks are often massively overparameterized leading to high compute costs during training and at inference. One effective method to improve both the compute and energy efficiency of neural networks while maintaining good performance is structured pruning, where full network structures (e.g.~neurons or convolutional filters) that have limited impact on the model output are removed. In this work, we propose Bayesian Model Reduction for Structured pruning (BMRS), a fully end-to-end Bayesian method of structured pruning. BMRS is based on two recent methods: Bayesian structured pruning with multiplicative noise, and Bayesian model reduction (BMR), a method which allows efficient comparison of Bayesian models under a change in prior. We present two realizations of BMRS derived from different priors which yield different structured pruning characteristics: 1) BMRS_N with the truncated log-normal prior, which offers reliable compression rates and accuracy without the need for tuning any thresholds and 2) BMRS_U with the truncated log-uniform prior that can achieve more aggressive compression based on the boundaries of truncation. Overall, we find that BMRS offers a theoretically grounded approach to structured pruning of neural networks yielding both high compression rates and accuracy. Experiments on multiple datasets and neural networks of varying complexity showed that the two BMRS methods offer a competitive performance-efficiency trade-off compared to other pruning methods.