Data to 14 Oct 2026

Dropout Training as Adaptive Regularization

arXiv 1307.1493 ↗First posted 4 Jul 2013stat.ML cs.LG stat.MEcited by 93

Not on the map. Why?

Abstract (arXiv)

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Jul 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 19; 2023: 15; 2024: 9; 2025: 17; 2026: 20.

93 papers.

Sort:

By area:

2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 43
2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 31
2022DropMessage: Unifying Random Dropping for Graph Neural Networkscited by 29
2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
2022A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspectivecited by 27
2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
2022Implicit regularization of dropoutcited by 25
2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
2021Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machinescited by 22
2021Convergence and Alignment of Gradient Descent with Random Backpropagation Weightscited by 14

38 more

2022Domain Adaptation under Missingness Shiftcited by 13
2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
2023Generalized equivalences between subsampling and ridge regularizationcited by 10
2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
2021Structured Dropout Variational Inference for Bayesian Neural Networkscited by 7
2021Regularization in ResNet with Stochastic Depthcited by 7
2022Over-the-Air Split Machine Learning in Wireless MIMO Networkscited by 7
2024Singular-limit analysis of gradient descent with noise injectioncited by 7
2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
2021The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularizationcited by 6
2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
2022Over-the-Air Federated Learning with Privacy Protection via Correlated Additive Perturbationscited by 5
2022A Survey on Dropout Methods and Experimental Verification in Recommendationcited by 4
2022Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Tracecited by 4
2022Data-Efficient Augmentation for Training Neural Networkscited by 4
2022Data-driven Science and Machine Learning Methods in Laser-Plasma Physicscited by 4
2024Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglementcited by 4
2025Random Forest Autoencoders for Guided Representation Learningcited by 4
2025Drop Dropout on Single-Epoch Language Model Pretrainingcited by 4
2022Recurrent Encoder-Decoder Networks for Vessel Trajectory Prediction with Uncertainty Estimationcited by 3
2022Invariant Structure Learning for Better Generalization and Causal Explainabilitycited by 3
2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
2025A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalizationcited by 3
2025In-Context Learning Enhanced Credibility Transformercited by 3
2025Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regimecited by 3
2026DropoutTS: Sample-Adaptive Dropout for Robust Time Series Forecastingcited by 3
2021Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learningcited by 2
2021PLACE dropout: A Progressive Layer-wise and Channel-wise Dropout for Domain Generalizationcited by 2
2022Frequency Dropout: Feature-Level Regularization via Randomized Filteringcited by 2
2022Noise Injection as a Probe of Deep Learning Dynamicscited by 2
2023Dropout Drops Double Descentcited by 2
2023Weight Freezing: A Regularization Approach for Fully Connected Layers with an Application in EEG Classificationcited by 2
2023Exploration and Exploitation of Unlabeled Data for Open-Set Semi-Supervised Learningcited by 2
2023AdaFlood: Adaptive Flood Regularizationcited by 2
2024Improving Generalization in Aerial and Terrestrial Mobile Robots Control Through Delayed Policy Learningcited by 2
2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2

50 most cited of 93 All 93 (JSON)