mlsanityData to 7 Oct 2026

The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization

arXiv 2106.07769 ↗First posted 14 Jun 2021cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Among the most successful methods for sparsifying deep (neural) networks are those that adaptively mask the network weights throughout training. By examining this masking, or dropout, in the linear case, we uncover a duality between such adaptive methods and regularization through the so-called "η-trick" that casts both as iteratively reweighted optimizations. We show that any dropout strategy that adapts to the weights in a monotonic way corresponds to an effective subquadratic regularization penalty, and therefore leads to sparse solutions. We obtain the effective penalties for several popular sparsification strategies, which are remarkably similar to classical penalties commonly used in sparse optimization. Considering variational dropout as a case study, we demonstrate similar empirical behavior between the adaptive dropout method and classical methods on the task of deep network sparsification, validating our theory.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 47 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

23 of 47 references.

References · 23

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  8. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  9. 2019The State of Sparsity in Deep Neural Networkscited by 471
  10. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  11. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  12. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
11 more
  1. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  2. 2013Dropout Training as Adaptive Regularizationcited by 89
  3. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  4. 2015De-biasing the Lasso: Optimal Sample Size for Gaussian Designscited by 35
  5. 2020On the proliferation of support vectors in high dimensionscited by 32
  6. 2013Stochastic Majorization-Minimization Algorithms for Large-Scale Optimizationcited by 28
  7. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  8. 2018Variational Bayesian dropout: pitfalls and fixescited by 22
  9. 2020On Convergence and Generalization of Dropout Trainingcited by 19
  10. 2018False Discovery Rate Control via Debiased Lassocited by 8
  11. 2017Design of a Very Compact CNN Classifier for Online Handwritten Chinese Character Recognition Using DropWeight and Global Poolingcited by 5

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 2; 2024: 2.

6 papers.

  1. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  2. 2021Prune Your Model Before Distill Itcited by 12
  3. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  4. 2023Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuningcited by 11
  5. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  6. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1