# On the Implicit Bias of Dropout

arXiv: https://arxiv.org/abs/1806.09777 · HTML: https://mlsanity.com/paper/1806.09777/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2018.

## Built on by (most cited first, 22 of 22)

- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [The dynamics of representation learning in shallow, non-linear autoencoders](https://mlsanity.com/paper/2201.02115/) (2022)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective](https://mlsanity.com/paper/2210.05021/) (2022)
- [A Unified Approach to Controlling Implicit Regularization via Mirror Descent](https://mlsanity.com/paper/2306.13853/) (2023)
- [An Overview of Low-Rank Structures in the Training and Adaptation of Large Models](https://mlsanity.com/paper/2503.19859/) (2025)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://mlsanity.com/paper/2602.15322/) (2026)
- [Weight Expansion: A New Perspective on Dropout and Generalization](https://mlsanity.com/paper/2201.09209/) (2022)
- [An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network](https://mlsanity.com/paper/2105.11675/) (2021)
- [Analytic theory of dropout regularization](https://mlsanity.com/paper/2505.07792/) (2025)
- [Dropout Neural Network Training Viewed from a Percolation Perspective](https://mlsanity.com/paper/2512.13853/) (2025)
- [Overlap-Adaptive Regularization for Conditional Average Treatment Effect Estimation](https://mlsanity.com/paper/2509.24962/) (2025)
- [Estimating Implicit Regularization in Deep Learning](https://mlsanity.com/paper/2605.05436/) (2026)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [A Reflection on Learning from Data: Epistemology Issues and Limitations](https://mlsanity.com/paper/2107.13270/) (2021)
- [Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provably](https://mlsanity.com/paper/2202.03535/) (2022)
- [ShuffleMix: Improving Representations via Channel-Wise Shuffle of Interpolated Hidden States](https://mlsanity.com/paper/2305.18684/) (2023)
- [Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets](https://mlsanity.com/paper/2607.16761/) (2026)
- [Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox](https://mlsanity.com/paper/2609.32060/) (2026)

## Abstract (arXiv)

Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. In addition, we provide a complete characterization of the optimization landscape induced by dropout.