mlsanityData to 7 Oct 2026

Implicit regularization of dropout

arXiv 2207.05952 ↗First posted 13 Jul 2022cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we present a theoretical derivation of an implicit regularization of dropout, which is validated by a series of experiments. Additionally, we numerically study two implications of the implicit regularization, which intuitively rationalizes why dropout helps generalization. Firstly, we find that input weights of hidden neurons tend to condense on isolated orientations trained with dropout. Condensation is a feature in the non-linear learning process, which makes the network less complex. Secondly, we experimentally find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training, and the implicit regularization is the key to finding flat solutions. Although our theory mainly focuses on dropout used in the last hidden layer, our experiments apply to general dropout in training neural networks. This work points out a distinct characteristic of dropout compared with stochastic gradient descent and serves as an important basis for fully understanding dropout.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

44 of 61 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
32 more
  1. 2018Training behavior of deep neural network in frequency domaincited by 174
  2. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  3. 2017Generalization in Deep Learningcited by 146
  4. 2020Implicit Gradient Regularizationcited by 145
  5. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  6. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  7. 2013Dropout Training as Adaptive Regularizationcited by 89
  8. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  9. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  10. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  11. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  12. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  13. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  14. 2019Theory of the Frequency Principle for General Deep Neural Networkscited by 45
  15. 2013A PAC-Bayesian Tutorial with A Dropout Boundcited by 43
  16. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  17. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  18. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  19. 2020Dropout as an Implicit Gating Mechanism For Continual Learningcited by 24
  20. 2018On the Implicit Bias of Dropoutcited by 22
  21. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  22. 2014On the Inductive Bias of Dropoutcited by 19
  23. 2020On Convergence and Generalization of Dropout Trainingcited by 19
  24. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  25. 2017Dropout as a Low-Rank Regularizer for Matrix Factorizationcited by 14
  26. 2020An analytic theory of shallow networks dynamics for hinge loss classificationcited by 14
  27. 2021Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networkscited by 14
  28. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  29. 2019On the Regularization Properties of Structured Dropoutcited by 5
  30. 2020Almost Sure Convergence of Dropout Algorithms for Neural Networkscited by 4
  31. 2020Dropout as a Regularizer of Interaction Effectscited by 4
  32. 2014Altitude Training: Strong Bounds for Single-Layer Dropoutcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 6; 2025: 9; 2026: 3.

24 papers.

Sort:

By area:

  1. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  2. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  3. 2025An overview of condensation phenomenon in deep learningcited by 13
  4. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  5. 2022Linear Stability Hypothesis and Rank Stratification for Nonlinear Modelscited by 7
  6. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 7
  7. 2025How Effective Can Dropout Be in Multiple Instance Learning ?cited by 7
  8. 2023Optimistic Estimate Uncovers the Potential of Nonlinear Modelscited by 5
  9. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  10. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 5
  11. 2024X-SHIELD: Regularization for eXplainable Artificial Intelligencecited by 3
  12. 2025Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regimecited by 3
12 more
  1. 2025FIC-TSC: Learning Time Series Classification with Fisher Information Constraintcited by 2
  2. 2025Analytic theory of dropout regularizationcited by 2
  3. 2024Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?cited by 1
  4. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  5. 2023Feature Noise Boosts DNN Generalization under Label Noisecited by 0
  6. 2024Loss Jump During Loss Switch in Solving PDEs with Neural Networkscited by 0
  7. 2024Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterizationcited by 0
  8. 2025High-order Regularization for Machine Learning and Learning-based Controlcited by 0
  9. 2025KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approachescited by 0
  10. 2025Convergence, design and training of continuous-time dropout as a random batch methodcited by 0
  11. 2026DPD-Cancer: Explainable Graph-Based Deep Learning for Small Molecule Anti-Cancer Activity Predictioncited by 0
  12. 2026Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNetscited by 0