# Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee

arXiv: https://arxiv.org/abs/2011.07439 · HTML: https://mlsanity.com/paper/2011.07439/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2020.

## Built on by (most cited first, 30 of 37)

- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants](https://mlsanity.com/paper/2211.10575/) (2022)
- [Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruning](https://mlsanity.com/paper/2212.03537/) (2022)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Posterior and variational inference for deep neural networks with heavy-tailed weights](https://mlsanity.com/paper/2406.03369/) (2024)
- [Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference](https://mlsanity.com/paper/2305.14765/) (2023)
- [FedHB: Hierarchical Bayesian Federated Learning](https://mlsanity.com/paper/2305.04979/) (2023)
- [Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning](https://mlsanity.com/paper/2102.12328/) (2021)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Misclassification bounds for PAC-Bayesian sparse deep learning](https://mlsanity.com/paper/2405.01304/) (2024)
- [Sequential Bayesian Neural Subnetwork Ensembles](https://mlsanity.com/paper/2206.00794/) (2022)
- [Federated Learning via Variational Bayesian Inference: Personalization, Sparsity and Clustering](https://mlsanity.com/paper/2303.04345/) (2023)
- [Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty](https://mlsanity.com/paper/2305.00934/) (2023)
- [Deep Horseshoe Gaussian Processes](https://mlsanity.com/paper/2403.01737/) (2024)
- [Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks](https://mlsanity.com/paper/2308.09104/) (2023)
- [FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification](https://mlsanity.com/paper/2404.15657/) (2024)
- [Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Data](https://mlsanity.com/paper/2202.09463/) (2022)
- [On the optimization and pruning for Bayesian deep learning](https://mlsanity.com/paper/2210.12957/) (2022)
- [Sparsifying Bayesian neural networks with latent binary variables and normalizing flows](https://mlsanity.com/paper/2305.03395/) (2023)
- [Bayesian Personalized Federated Learning with Shared and Personalized Uncertainty Representations](https://mlsanity.com/paper/2309.15499/) (2023)
- [BMRS: Bayesian Model Reduction for Structured Pruning](https://mlsanity.com/paper/2406.01345/) (2024)
- [Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality](https://mlsanity.com/paper/2506.19144/) (2025)
- [Masked Bayesian Neural Networks : Computation and Optimality](https://mlsanity.com/paper/2206.00853/) (2022)
- [Projective Integral Updates for High-Dimensional Variational Inference](https://mlsanity.com/paper/2301.08374/) (2023)
- [Personalized Federated Learning via Amortized Bayesian Meta-Learning](https://mlsanity.com/paper/2307.02222/) (2023)
- [Bayesian sparsification for deep neural networks with Bayesian model reduction](https://mlsanity.com/paper/2309.12095/) (2023)
- [Bayesian Computation in Deep Learning](https://mlsanity.com/paper/2502.18300/) (2025)
- [Explainable Bayesian deep learning through input-skip Latent Binary Bayesian Neural Networks](https://mlsanity.com/paper/2503.10496/) (2025)
- [Bayesian Coreset Optimization for Personalized Federated Learning](https://mlsanity.com/paper/2511.01800/) (2025)
- [A Hierarchical Bayesian Model for Deep Few-Shot Meta Learning](https://mlsanity.com/paper/2306.09702/) (2023)

## Abstract (arXiv)

Sparse deep learning aims to address the challenge of huge storage consumption by deep neural networks, and to recover the sparse structure of target functions. Although tremendous empirical successes have been achieved, most sparse deep learning algorithms are lacking of theoretical support. On the other hand, another line of works have proposed theoretical frameworks that are computationally infeasible. In this paper, we train sparse deep neural networks with a fully Bayesian treatment under spike-and-slab priors, and develop a set of computationally efficient variational inferences via continuous relaxation of Bernoulli distribution. The variational posterior contraction rate is provided, which justifies the consistency of the proposed variational Bayes method. Notably, our empirical results demonstrate that this variational procedure provides uncertainty quantification in terms of Bayesian predictive distribution and is also capable to accomplish consistent variable selection by training a sparse multi-layer neural network.