# On the Inductive Bias of Dropout

arXiv: https://arxiv.org/abs/1412.4736 · HTML: https://mlsanity.com/paper/1412.4736/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2014.

## Built on by (most cited first, 19 of 19)

- [FairDrop: Biased Edge Dropout for Enhancing Fairness in Graph Representation Learning](https://mlsanity.com/paper/2104.14210/) (2021)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Equivariance and Invariance Inductive Bias for Learning from Insufficient Data](https://mlsanity.com/paper/2207.12258/) (2022)
- [Stochastic Modified Equations and Dynamics of Dropout Algorithm](https://mlsanity.com/paper/2305.15850/) (2023)
- [Efficient Stagewise Pretraining via Progressive Subnetworks](https://mlsanity.com/paper/2402.05913/) (2024)
- [Sparse-penalized deep neural networks estimator under weak dependence](https://mlsanity.com/paper/2303.01406/) (2023)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Regularization in ResNet with Stochastic Depth](https://mlsanity.com/paper/2106.03091/) (2021)
- [Weight Expansion: A New Perspective on Dropout and Generalization](https://mlsanity.com/paper/2201.09209/) (2022)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace](https://mlsanity.com/paper/2208.05924/) (2022)
- [What do Large Language Models Learn beyond Language?](https://mlsanity.com/paper/2210.12302/) (2022)
- [Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime](https://mlsanity.com/paper/2510.07554/) (2025)
- [Information Geometry of Dropout Training](https://mlsanity.com/paper/2206.10936/) (2022)
- [Dropout Drops Double Descent](https://mlsanity.com/paper/2305.16179/) (2023)
- [Estimating Implicit Regularization in Deep Learning](https://mlsanity.com/paper/2605.05436/) (2026)
- [Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provably](https://mlsanity.com/paper/2202.03535/) (2022)
- [Dropout Regularization in Extended Generalized Linear Models based on Double Exponential Families](https://mlsanity.com/paper/2305.06625/) (2023)
- [Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets](https://mlsanity.com/paper/2607.16761/) (2026)

## Abstract (arXiv)

Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout should be applied and how to use it most effectively. In this paper we continue the exploration of dropout as a regularizer pioneered by Wager, et.al. We focus on linear classification where a convex proxy to the misclassification loss (i.e. the logistic loss used in logistic regression) is minimized. We show: (a) when the dropout-regularized criterion has a unique minimizer, (b) when the dropout-regularization penalty goes to infinity with the weights, and when it remains bounded, (c) that the dropout regularization can be non-monotonic as individual weights increase from 0, and (d) that the dropout regularization penalty may not be convex. This last point is particularly surprising because the combination of dropout regularization with any convex loss proxy is always a convex function. In order to contrast dropout regularization with L2 regularization, we formalize the notion of when different sources are more compatible with different regularizers. We then exhibit distributions that are provably more compatible with dropout regularization than L2 regularization, and vice versa. These sources provide additional insight into how the inductive biases of dropout and L2 regularization differ. We provide some similar results for L1 regularization.