# Dropout: Explicit Forms and Capacity Control

arXiv: https://arxiv.org/abs/2003.03397 · HTML: https://mlsanity.com/paper/2003.03397/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2020.

## Built on by (most cited first, 27 of 27)

- [Meta-Learning with Fewer Tasks through Task Interpolation](https://mlsanity.com/paper/2106.02695/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2209.08928/) (2022)
- [A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective](https://mlsanity.com/paper/2208.09913/) (2022)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines](https://mlsanity.com/paper/2106.01506/) (2021)
- [Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers](https://mlsanity.com/paper/2205.14336/) (2022)
- [Efficient Stagewise Pretraining via Progressive Subnetworks](https://mlsanity.com/paper/2402.05913/) (2024)
- [An Overview of Low-Rank Structures in the Training and Adaptation of Large Models](https://mlsanity.com/paper/2503.19859/) (2025)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization](https://mlsanity.com/paper/2106.07769/) (2021)
- [Towards Size-Independent Generalization Bounds for Deep Operator Nets](https://mlsanity.com/paper/2205.11359/) (2022)
- [What training reveals about neural network complexity](https://mlsanity.com/paper/2106.04186/) (2021)
- [Stochastic Neural Networks with Infinite Width are Deterministic](https://mlsanity.com/paper/2201.12724/) (2022)
- [Boosting Factorization Machines via Saliency-Guided Mixup](https://mlsanity.com/paper/2206.08661/) (2022)
- [Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime](https://mlsanity.com/paper/2510.07554/) (2025)
- [Generalization of GANs and overparameterized models under Lipschitz continuity](https://mlsanity.com/paper/2104.02388/) (2021)
- [Reweighted Mixup for Subpopulation Shift](https://mlsanity.com/paper/2304.04148/) (2023)
- [Dropout Drops Double Descent](https://mlsanity.com/paper/2305.16179/) (2023)
- [Analytic theory of dropout regularization](https://mlsanity.com/paper/2505.07792/) (2025)
- [Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction](https://mlsanity.com/paper/2206.00913/) (2022)
- [LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation](https://mlsanity.com/paper/2505.19659/) (2025)
- [Overlap-Adaptive Regularization for Conditional Average Treatment Effect Estimation](https://mlsanity.com/paper/2509.24962/) (2025)
- [The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Control](https://mlsanity.com/paper/2511.00958/) (2025)
- [Explicit Dropout: Deterministic Regularization for Transformer Architectures](https://mlsanity.com/paper/2604.20505/) (2026)
- [Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets](https://mlsanity.com/paper/2607.16761/) (2026)

## Abstract (arXiv)

We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.