# Convergence Rates of Variational Inference in Sparse Deep Learning

arXiv: https://arxiv.org/abs/1908.04847 · HTML: https://mlsanity.com/paper/1908.04847/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2019.

## Built on by (most cited first, 27 of 27)

- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Posterior and variational inference for deep neural networks with heavy-tailed weights](https://mlsanity.com/paper/2406.03369/) (2024)
- [Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference](https://mlsanity.com/paper/2305.14765/) (2023)
- [FedHB: Hierarchical Bayesian Federated Learning](https://mlsanity.com/paper/2305.04979/) (2023)
- [On the inability of Gaussian process regression to optimally learn compositional functions](https://mlsanity.com/paper/2205.07764/) (2022)
- [Asymptotic Properties for Bayesian Neural Network in Besov Space](https://mlsanity.com/paper/2206.00241/) (2022)
- [Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection](https://mlsanity.com/paper/2207.11208/) (2022)
- [Misclassification bounds for PAC-Bayesian sparse deep learning](https://mlsanity.com/paper/2405.01304/) (2024)
- [Federated Learning via Variational Bayesian Inference: Personalization, Sparsity and Clustering](https://mlsanity.com/paper/2303.04345/) (2023)
- [Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty](https://mlsanity.com/paper/2305.00934/) (2023)
- [Deep Horseshoe Gaussian Processes](https://mlsanity.com/paper/2403.01737/) (2024)
- [FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout](https://mlsanity.com/paper/2307.07172/) (2023)
- [Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks](https://mlsanity.com/paper/2308.09104/) (2023)
- [Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds](https://mlsanity.com/paper/2410.21702/) (2024)
- [A fast asynchronous MCMC sampler for sparse Bayesian inference](https://mlsanity.com/paper/2108.06446/) (2021)
- [Optimization Guarantees for Square-Root Natural-Gradient Variational Inference](https://mlsanity.com/paper/2507.07853/) (2025)
- [A PAC-Bayes oracle inequality for sparse neural networks](https://mlsanity.com/paper/2204.12392/) (2022)
- [Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality](https://mlsanity.com/paper/2506.19144/) (2025)
- [Bayesian Lottery Ticket Hypothesis](https://mlsanity.com/paper/2602.18825/) (2026)
- [Masked Bayesian Neural Networks : Computation and Optimality](https://mlsanity.com/paper/2206.00853/) (2022)
- [Personalized Federated Learning via Amortized Bayesian Meta-Learning](https://mlsanity.com/paper/2307.02222/) (2023)
- [The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networks](https://mlsanity.com/paper/2310.09335/) (2023)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions](https://mlsanity.com/paper/2510.08999/) (2025)
- [Posterior Concentration of Bayesian Physics-Informed Neural Networks for Elliptic PDEs](https://mlsanity.com/paper/2605.08672/) (2026)

## Abstract (arXiv)

Variational inference is becoming more and more popular for approximating intractable posterior distributions in Bayesian statistics and machine learning. Meanwhile, a few recent works have provided theoretical justification and new insights on deep neural networks for estimating smooth functions in usual settings such as nonparametric regression. In this paper, we show that variational inference for sparse deep learning retains the same generalization properties than exact Bayesian inference. In particular, we highlight the connection between estimation and approximation theories via the classical bias-variance trade-off and show that it leads to near-minimax rates of convergence for H"older smooth functions. Additionally, we show that the model selection framework over the neural network architecture via ELBO maximization does not overfit and adaptively achieves the optimal rate of convergence.