mlsanityData to 7 Oct 2026

Variational Dropout and the Local Reparameterization Trick

arXiv 1506.02557 ↗First posted 8 Jun 2015stat.ML cs.LG stat.COcited by 489

Not on the map. Why?

Abstract (arXiv)

We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the minibatch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.

Key facts. Cited by 489 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 1; 2021: 101; 2022: 100; 2023: 103; 2024: 78; 2025: 72; 2026: 31.

489 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  4. 2019Causal Confusion in Imitation Learningcited by 286
  5. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  6. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  7. 2022ActiveNeRF: Learning where to See with Uncertainty Estimationcited by 124
  8. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  9. 2021Bayesian Neural Network Priors Revisitedcited by 121
  10. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  11. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  12. 2021Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approachcited by 105
38 more
  1. 2022CARD: Classification and Regression Diffusion Modelscited by 102
  2. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  3. 2021Variational Transformer Networks for Layout Generationcited by 88
  4. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 83
  5. 2021Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representationscited by 80
  6. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  7. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  8. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  9. 2023A Review of Uncertainty Estimation and its Application in Medical Imagingcited by 59
  10. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  11. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  12. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  13. 2023Learning to Maximize Mutual Information for Dynamic Feature Selectioncited by 54
  14. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  15. 2021Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibrationcited by 48
  16. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  17. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  18. 2025Reinforced Latent Reasoning for LLM-based Recommendationcited by 46
  19. 2021Posterior Meta-Replay for Continual Learningcited by 45
  20. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  21. 2021LiBRe: A Practical Bayesian Approach to Adversarial Detectioncited by 40
  22. 2024Neural Network Diffusioncited by 39
  23. 2026ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrievalcited by 39
  24. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  25. 2022Probabilistic Representations for Video Contrastive Learningcited by 33
  26. 2022Geometry Aligned Variational Transformer for Image-conditioned Layout Generationcited by 33
  27. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  28. 2021NoiseGrad: Enhancing Explanations by Introducing Stochasticity to Model Weightscited by 28
  29. 2022Uncertainty-Aware Learning Against Label Noise on Imbalanced Datasetscited by 28
  30. 2022Robust Information Bottleneck for Task-Oriented Communication with Digital Modulationcited by 28
  31. 2022Copula Conformal Prediction for Multi-step Time Series Forecastingcited by 28
  32. 2021A Bit More Bayesian: Domain-Invariant Learning with Uncertaintycited by 27
  33. 2021Deep Ensembles from a Bayesian Perspectivecited by 27
  34. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  35. 2021Selfish Sparse RNN Trainingcited by 26
  36. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  37. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  38. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26

50 most cited of 489 All 489 (JSON)