# On Convergence and Generalization of Dropout Training

arXiv: https://arxiv.org/abs/2010.12711 · HTML: https://mlsanity.com/paper/2010.12711/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate](https://mlsanity.com/paper/2402.02769/) (2024)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons](https://mlsanity.com/paper/2112.02668/) (2021)
- [Stochastic Modified Equations and Dynamics of Dropout Algorithm](https://mlsanity.com/paper/2305.15850/) (2023)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization](https://mlsanity.com/paper/2106.07769/) (2021)
- [Weight Expansion: A New Perspective on Dropout and Generalization](https://mlsanity.com/paper/2201.09209/) (2022)
- [What training reveals about neural network complexity](https://mlsanity.com/paper/2106.04186/) (2021)
- [Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime](https://mlsanity.com/paper/2510.07554/) (2025)
- [Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition](https://mlsanity.com/paper/2103.16889/) (2021)
- [Generalization of GANs and overparameterized models under Lipschitz continuity](https://mlsanity.com/paper/2104.02388/) (2021)
- [Dropout in Training Neural Networks: Flatness of Solution and Noise Structure](https://mlsanity.com/paper/2111.01022/) (2021)
- [Robust Collaborative Inference with Vertically Split Data Over Dynamic Device Environments](https://mlsanity.com/paper/2312.16638/) (2023)
- [Dropout Neural Network Training Viewed from a Percolation Perspective](https://mlsanity.com/paper/2512.13853/) (2025)
- [Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions](https://mlsanity.com/paper/2206.03254/) (2022)
- [Convergence, design and training of continuous-time dropout as a random batch method](https://mlsanity.com/paper/2510.13134/) (2025)
- [Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets](https://mlsanity.com/paper/2607.16761/) (2026)

## Abstract (arXiv)

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations. Under mild overparametrization and assuming that the limiting kernel can separate the data distribution with a positive margin, we show that dropout training with logistic loss achieves ε-suboptimality in test error in O(1/ε) iterations.