# Analytic theory of dropout regularization

arXiv: https://arxiv.org/abs/2505.07792 · HTML: https://mlsanity.com/paper/2505.07792/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 14 of 14)

- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019)
- [Dropout Training as Adaptive Regularization](https://mlsanity.com/paper/1307.1493/) (2013)
- [Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification](https://mlsanity.com/paper/2006.06098/) (2020)
- [The committee machine: Computational to statistical gaps in learning a two-layers neural network](https://mlsanity.com/paper/1806.05451/) (2018)
- [Curriculum Dropout](https://mlsanity.com/paper/1703.06229/) (2017)
- [Dropout: Explicit Forms and Capacity Control](https://mlsanity.com/paper/2003.03397/) (2020)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [On the Implicit Bias of Dropout](https://mlsanity.com/paper/1806.09777/) (2018)
- [Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation](https://mlsanity.com/paper/1910.07476/) (2019)
- [Optimal Protocols for Continual Learning via Statistical Physics and Control Theory](https://mlsanity.com/paper/2409.18061/) (2024)
- [Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines](https://mlsanity.com/paper/2104.14546/) (2021)
- [A Theory of Initialisation's Impact on Specialisation](https://mlsanity.com/paper/2503.02526/) (2025)
- [On-line learning dynamics of ReLU neural networks using statistical physics techniques](https://mlsanity.com/paper/1903.07378/) (2019)

## Built on by (most cited first, 2 of 2)

- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [An Analytical Theory of Auxiliary Learning](https://mlsanity.com/paper/2609.29774/) (2026)

## Abstract (arXiv)

Dropout is a regularization technique widely used in training artificial neural networks to mitigate overfitting. It consists of dynamically deactivating subsets of the network during training to promote more robust representations. Despite its widespread adoption, dropout probabilities are often selected heuristically, and theoretical explanations of its success remain sparse. Here, we analytically study dropout in two-layer neural networks trained with online stochastic gradient descent. In the high-dimensional limit, we derive a set of ordinary differential equations that fully characterize the evolution of the network during training and capture the effects of dropout. We obtain a number of exact results describing the generalization error and the optimal dropout probability at short, intermediate, and long training times. Our analysis shows that dropout reduces detrimental correlations between hidden nodes, mitigates the impact of label noise, and that the optimal dropout probability increases with the level of noise in the data. Our results are validated by extensive numerical simulations.