mlsanityData to 7 Oct 2026

LocalDrop: A Hybrid Regularization for Deep Neural Networks

arXiv 2103.00719 ↗First posted 1 Mar 2021cs.LG cs.AI stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs), including drop rates and weight matrices, has been developed based on the proposed upper bound of the local Rademacher complexity by the strict mathematical deduction. The analyses of dropout in FCNs and DropBlock in CNNs with keep rate matrices in different layers are also included in the complexity analyses. With the new regularization function, we establish a two-stage procedure to obtain the optimal keep rate matrix and weight matrix to realize the whole training model. Extensive experiments have been conducted to demonstrate the effectiveness of LocalDrop in different models by comparing it with several algorithms and the effects of different hyperparameters on the final performances.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

27 of 48 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2016Wide Residual Networkscited by 3,057
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  10. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  11. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  12. 2015Norm-Based Capacity Control in Neural Networkscited by 299
15 more
  1. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  2. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  3. 2018DropBlock: A regularization method for convolutional networkscited by 261
  4. 2017Concrete Dropoutcited by 227
  5. 2014Efficient Object Localization Using Convolutional Networkscited by 226
  6. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  7. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  8. 2017Adversarial Dropout for Supervised and Semi-supervised Learningcited by 38
  9. 2014Fractional Max-Poolingcited by 35
  10. 2018On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyondcited by 24
  11. 2018Variational Bayesian dropout: pitfalls and fixescited by 22
  12. 2014Dropout Rademacher Complexity of Deep Neural Networkscited by 14
  13. 2018Capacity Control of ReLU Neural Networks by Basis-path Normcited by 10
  14. 2015A Spatial Layout and Scale Invariant Feature Representation for Indoor Scene Classificationcited by 5
  15. 2017MAT: A Multimodal Attentive Translator for Image Captioningcited by 4

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 1; 2025: 1.

8 papers.

  1. 2022Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networkscited by 55
  2. 2022A Survey on Dropout Methods and Experimental Verification in Recommendationcited by 4
  3. 2021Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networkscited by 0
  4. 2022MaxDropoutV2: An Improved Method to Drop out Neurons in Convolutional Neural Networkscited by 0
  5. 2023Regularization Through Simultaneous Learning: A Case Study on Plant Classificationcited by 0
  6. 2023R-Block: Regularized Block of Dropout for convolutional networkscited by 0
  7. 2024Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognitioncited by 0
  8. 2025Sharper Risk Bound for Multi-Task Learning with Multi-Graph Dependent Datacited by 0