# Stochastic Weight Sharing for Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2505.17856 · HTML: https://mlsanity.com/paper/2505.17856/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 70 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Expectation Propagation for approximate Bayesian inference](https://mlsanity.com/paper/1301.2294/) (2013)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Soft Weight-Sharing for Neural Network Compression](https://mlsanity.com/paper/1702.04008/) (2017)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [A Survey on Methods and Theories of Quantized Neural Networks](https://mlsanity.com/paper/1808.04752/) (2018)
- [Faster Wasserstein Distance Estimation with the Sinkhorn Divergence](https://mlsanity.com/paper/2006.08172/) (2020)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Searching for Low-Bit Weights in Quantized Neural Networks](https://mlsanity.com/paper/2009.08695/) (2020)
- [Bayesian Inference for Large Scale Image Classification](https://mlsanity.com/paper/1908.03491/) (2019)
- [Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification](https://mlsanity.com/paper/2012.02818/) (2020)
- [Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models](https://mlsanity.com/paper/2312.15297/) (2023)
- [Flat Seeking Bayesian Neural Networks](https://mlsanity.com/paper/2302.02713/) (2023)
- [Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks](https://mlsanity.com/paper/2112.15139/) (2021)
- [Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference](https://mlsanity.com/paper/2305.14765/) (2023)
- [On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/2102.11062/) (2021)
- [An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis](https://mlsanity.com/paper/2007.07129/) (2020)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization](https://mlsanity.com/paper/2210.09134/) (2022)
- [Vector Quantized Bayesian Neural Network Inference for Data Streams](https://mlsanity.com/paper/1907.05911/) (2019)
- [Probabilistic Reach-Avoid for Bayesian Neural Networks](https://mlsanity.com/paper/2310.01951/) (2023)
- [Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study](https://mlsanity.com/paper/2005.11619/) (2020)
- [Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning](https://mlsanity.com/paper/2407.02721/) (2024)

## Built on by (most cited first, 2 of 2)

- [Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2601.21003/) (2026)
- [SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions](https://mlsanity.com/paper/2510.08999/) (2025)

## Abstract (arXiv)

While offering a principled framework for uncertainty quantification in deep learning, the employment of Bayesian Neural Networks (BNNs) is still constrained by their increased computational requirements and the convergence difficulties when training very deep, state-of-the-art architectures. In this work, we reinterpret weight-sharing quantization techniques from a stochastic perspective in the context of training and inference with Bayesian Neural Networks (BNNs). Specifically, we leverage 2D adaptive Gaussian distributions, Wasserstein distance estimations, and alpha blending to encode the stochastic behaviour of a BNN in a lower dimensional, soft Gaussian representation. Through extensive empirical investigation, we demonstrate that our approach significantly reduces the computational overhead inherent in Bayesian learning by several orders of magnitude, enabling the efficient Bayesian training of large-scale models, such as ResNet-101 and Vision Transformer (VIT). On various computer vision benchmarks including CIFAR10, CIFAR100, and ImageNet1k. Our approach compresses model parameters by approximately 50x and reduces model size by 75, while achieving accuracy and uncertainty estimations comparable to the state-of-the-art.