# Adaptive Network Sparsification with Dependent Variational Beta-Bernoulli Dropout

arXiv: https://arxiv.org/abs/1805.10896 · HTML: https://mlsanity.com/paper/1805.10896/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2018.

## Built on by (most cited first, 3 of 3)

- [Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2208.01573/) (2022)
- [Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation](https://mlsanity.com/paper/2206.09604/) (2022)
- [SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences](https://mlsanity.com/paper/2208.02169/) (2022)

## Abstract (arXiv)

While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity-inducing beta Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsity-inducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.