# Variational Dropout and the Local Reparameterization Trick

arXiv: https://arxiv.org/abs/1506.02557 · HTML: https://mlsanity.com/paper/1506.02557/

## Key facts

- Cited by 490 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2015.

## Built on by (most cited first, 30 of 490)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [ActiveNeRF: Learning where to See with Uncertainty Estimation](https://mlsanity.com/paper/2209.08546/) (2022)
- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [CARD: Classification and Regression Diffusion Models](https://mlsanity.com/paper/2206.07275/) (2022)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [Do Differentiable Simulators Give Better Policy Gradients?](https://mlsanity.com/paper/2202.00817/) (2022)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations](https://mlsanity.com/paper/2109.02123/) (2021)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [A Review of Uncertainty Estimation and its Application in Medical Imaging](https://mlsanity.com/paper/2302.08119/) (2023)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [Learning to Maximize Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2301.00557/) (2023)
- [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://mlsanity.com/paper/2107.03356/) (2021)
- [Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification](https://mlsanity.com/paper/2303.08446/) (2023)
- [Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration](https://mlsanity.com/paper/2102.12182/) (2021)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [Differentiable Subset Pruning of Transformer Heads](https://mlsanity.com/paper/2108.04657/) (2021)
- [Reinforced Latent Reasoning for LLM-based Recommendation](https://mlsanity.com/paper/2505.19092/) (2025)

## Abstract (arXiv)

We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the minibatch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.