# Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization

arXiv: https://arxiv.org/abs/2210.09134 · HTML: https://mlsanity.com/paper/2210.09134/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids](https://mlsanity.com/paper/2005.11138/) (2020)
- [A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions](https://mlsanity.com/paper/2010.03954/) (2020)
- [Variational Bayesian dropout: pitfalls and fixes](https://mlsanity.com/paper/1807.01969/) (2018)
- [Bayesian model reduction](https://mlsanity.com/paper/1805.07092/) (2018)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Bayesian Compression for Natural Language Processing](https://mlsanity.com/paper/1810.10927/) (2018)
- [Restoring speech intelligibility for hearing aid users with deep learning](https://mlsanity.com/paper/2206.11567/) (2022)

## Built on by (most cited first, 6 of 6)

- [Toward Design of Synthetic Active Inference Agents by Mere Mortals](https://mlsanity.com/paper/2307.14145/) (2023)
- [BMRS: Bayesian Model Reduction for Structured Pruning](https://mlsanity.com/paper/2406.01345/) (2024)
- [Stochastic Weight Sharing for Bayesian Neural Networks](https://mlsanity.com/paper/2505.17856/) (2025)
- [Bayesian sparsification for deep neural networks with Bayesian model reduction](https://mlsanity.com/paper/2309.12095/) (2023)
- [Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery](https://mlsanity.com/paper/2411.09127/) (2024)
- [Improved Depth Estimation of Bayesian Neural Networks](https://mlsanity.com/paper/2410.10395/) (2024)

## Abstract (arXiv)

Bayesian model reduction provides an efficient approach for comparing the performance of all nested sub-models of a model, without re-evaluating any of these sub-models. Until now, Bayesian model reduction has been applied mainly in the computational neuroscience community on simple models. In this paper, we formulate and apply Bayesian model reduction to perform principled pruning of Bayesian neural networks, based on variational free energy minimization. Direct application of Bayesian model reduction, however, gives rise to approximation errors. Therefore, a novel iterative pruning algorithm is presented to alleviate the problems arising with naive Bayesian model reduction, as supported experimentally on the publicly available UCI datasets for different inference algorithms. This novel parameter pruning scheme solves the shortcomings of current state-of-the-art pruning methods that are used by the signal processing community. The proposed approach has a clear stopping criterion and minimizes the same objective that is used during training. Next to these benefits, our experiments indicate better model performance in comparison to state-of-the-art pruning schemes.