mlsanityData to 13 Oct 2026

The Implicit and Explicit Regularization Effects of Dropout

arXiv 2002.12915 ↗First posted 28 Feb 2020cs.LG stat.MLcited by 71

Not on the map. Why?

Abstract (arXiv)

Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work demonstrates that dropout introduces two distinct but entangled regularization effects: an explicit effect (also studied in prior work) which occurs since dropout modifies the expected training objective, and, perhaps surprisingly, an additional implicit effect from the stochasticity in the dropout training update. This implicit regularization effect is analogous to the effect of stochasticity in small mini-batch stochastic gradient descent. We disentangle these two effects through controlled experiments. We then derive analytic simplifications which characterize each effect in terms of the derivatives of the model and the loss, for deep neural networks. We demonstrate these simplified, analytic regularizers accurately capture the important aspects of dropout, showing they faithfully replace dropout in practice.

Key facts. Cited by 71 papers in the map (data through 13 Oct 2026). First posted on arXiv 28 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 14; 2022: 14; 2023: 19; 2024: 11; 2025: 10; 2026: 3.

71 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 531
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 262
  3. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  4. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  5. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  6. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  7. 2021Noisy Recurrent Neural Networkscited by 42
  8. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  9. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  10. 2023Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learningcited by 32
  11. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  12. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
38 more
  1. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 31
  2. 2022Implicit regularization of dropoutcited by 25
  3. 2024Optimal ablation for interpretabilitycited by 22
  4. 2024Why is SAM Robust to Label Noise?cited by 21
  5. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18
  6. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  7. 2022Towards Bridging the Performance Gaps of Joint Energy-based Modelscited by 15
  8. 2022A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Momentscited by 14
  9. 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 13
  10. 2021Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injectionscited by 12
  11. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  12. 2021UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Costcited by 11
  13. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  14. 2023Flat Seeking Bayesian Neural Networkscited by 10
  15. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  16. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 10
  17. 2023Learning Large Graph Property Prediction via Graph Segment Trainingcited by 9
  18. 2021Structured Dropout Variational Inference for Bayesian Neural Networkscited by 7
  19. 2021Regularization in ResNet with Stochastic Depthcited by 7
  20. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  21. 2021The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularizationcited by 6
  22. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  23. 2022Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergencecited by 6
  24. 2022Using Human Perception to Regularize Transfer Learningcited by 6
  25. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  26. 2024Singular-limit analysis of gradient descent with noise injectioncited by 6
  27. 2024On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipescited by 6
  28. 2021What training reveals about neural network complexitycited by 5
  29. 2023Optimal Transport Model Distributional Robustnesscited by 5
  30. 2022Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Tracecited by 4
  31. 2023Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networkscited by 4
  32. 2024Towards Unraveling and Improving Generalization in World Modelscited by 4
  33. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shapingcited by 4
  34. 2022Bridging Model-based Safety and Model-free Reinforcement Learning through System Identification of Low Dimensional Linear Modelscited by 3
  35. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  36. 2025Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regimecited by 3
  37. 2021Dropout in Training Neural Networks: Flatness of Solution and Noise Structurecited by 2
  38. 2023Dropout Drops Double Descentcited by 2

50 most cited of 71 All 71 (JSON)