# Variational Dropout via Empirical Bayes

arXiv: https://arxiv.org/abs/1811.00596 · HTML: https://mlsanity.com/paper/1811.00596/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 4 of 4)

- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [BARNN: A Bayesian Autoregressive and Recurrent Neural Network](https://mlsanity.com/paper/2501.18665/) (2025)
- [A Principled Bayesian Framework for Training Binary and Spiking Neural Networks](https://mlsanity.com/paper/2505.17962/) (2025)
- [Joint Model and Data Sparsification via the Marginal Likelihood](https://mlsanity.com/paper/2605.29908/) (2026)

## Abstract (arXiv)

We study the Automatic Relevance Determination procedure applied to deep neural networks. We show that ARD applied to Bayesian DNNs with Gaussian approximate posterior distributions leads to a variational bound similar to that of variational dropout, and in the case of a fixed dropout rate, objectives are exactly the same. Experimental results show that the two approaches yield comparable results in practice even when the dropout rates are trained. This leads to an alternative Bayesian interpretation of dropout and mitigates some of the theoretical issues that arise with the use of improper priors in the variational dropout model. Additionally, we explore the use of the hierarchical priors in ARD and show that it helps achieve higher sparsity for the same accuracy.