mlsanityData to 14 Oct 2026

Variational Dropout Sparsifies Deep Neural Networks

arXiv 1701.05369 ↗First posted 19 Jan 2017stat.ML cs.LGcited by 265

Not on the map. Why?

Abstract (arXiv)

We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.

Key facts. Cited by 265 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 72; 2022: 61; 2023: 42; 2024: 33; 2025: 36; 2026: 21.

265 papers.

Sort:

By area:

  1. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  2. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  3. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  4. 2021Bayesian Neural Network Priors Revisitedcited by 121
  5. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  6. 2021Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approachcited by 110
  7. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  8. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  9. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
  10. 2022Deep Differentiable Logic Gate Networkscited by 67
  11. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  12. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
38 more
  1. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  2. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  3. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  4. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  5. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  6. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  7. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  8. 2021Differentiable Subset Pruning of Transformer Headscited by 47
  9. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  10. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  11. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  12. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  13. 2021Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributionscited by 39
  14. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 39
  15. 2022Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharingcited by 35
  16. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  17. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  18. 2021NoiseGrad: Enhancing Explanations by Introducing Stochasticity to Model Weightscited by 28
  19. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  20. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  21. 2021Selfish Sparse RNN Trainingcited by 26
  22. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  23. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  24. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  25. 2024CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Taskscited by 26
  26. 2021An Information-Theoretic Justification for Model Pruningcited by 25
  27. 2022Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraintscited by 25
  28. 2023Unified Data-Free Compression: Pruning and Quantization without Fine-Tuningcited by 23
  29. 2023Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrievalcited by 23
  30. 2023Adversarial Bayesian Augmentation for Single-Source Domain Generalizationcited by 22
  31. 2021GNN is a Counter? Revisiting GNN for Question Answeringcited by 21
  32. 2023Dropout Reduces Underfittingcited by 21
  33. 2021Pruning Randomly Initialized Neural Networks with Iterative Randomizationcited by 20
  34. 2024The Landscape of Unfolding with Machine Learningcited by 20
  35. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  36. 2021Explaining Bayesian Neural Networkscited by 18
  37. 2023Calibrating Multimodal Learningcited by 17
  38. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 17

50 most cited of 265 All 265 (JSON)