# LocalDrop: A Hybrid Regularization for Deep Neural Networks

arXiv: https://arxiv.org/abs/2103.00719 · HTML: https://mlsanity.com/paper/2103.00719/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations](https://mlsanity.com/paper/1606.01305/) (2016)
- [Adversarial Dropout for Supervised and Semi-supervised Learning](https://mlsanity.com/paper/1707.03631/) (2017)
- [Fractional Max-Pooling](https://mlsanity.com/paper/1412.6071/) (2014)
- [On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond](https://mlsanity.com/paper/1806.05159/) (2018)
- [Variational Bayesian dropout: pitfalls and fixes](https://mlsanity.com/paper/1807.01969/) (2018)
- [Dropout Rademacher Complexity of Deep Neural Networks](https://mlsanity.com/paper/1402.3811/) (2014)
- [Capacity Control of ReLU Neural Networks by Basis-path Norm](https://mlsanity.com/paper/1809.07122/) (2018)
- [A Spatial Layout and Scale Invariant Feature Representation for Indoor Scene Classification](https://mlsanity.com/paper/1506.05532/) (2015)
- [MAT: A Multimodal Attentive Translator for Image Captioning](https://mlsanity.com/paper/1702.05658/) (2017)

## Built on by (most cited first, 9 of 9)

- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Dynamic Evolution of Complex Networks: A Reinforcement Learning Approach Applying Evolutionary Games to Community Structure](https://mlsanity.com/paper/2506.17925/) (2025)
- [Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks](https://mlsanity.com/paper/2106.09677/) (2021)
- [MaxDropoutV2: An Improved Method to Drop out Neurons in Convolutional Neural Networks](https://mlsanity.com/paper/2203.02740/) (2022)
- [Regularization Through Simultaneous Learning: A Case Study on Plant Classification](https://mlsanity.com/paper/2305.13447/) (2023)
- [R-Block: Regularized Block of Dropout for convolutional networks](https://mlsanity.com/paper/2307.15150/) (2023)
- [Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition](https://mlsanity.com/paper/2401.14034/) (2024)
- [Sharper Risk Bound for Multi-Task Learning with Multi-Graph Dependent Data](https://mlsanity.com/paper/2502.18167/) (2025)

## Abstract (arXiv)

In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs), including drop rates and weight matrices, has been developed based on the proposed upper bound of the local Rademacher complexity by the strict mathematical deduction. The analyses of dropout in FCNs and DropBlock in CNNs with keep rate matrices in different layers are also included in the complexity analyses. With the new regularization function, we establish a two-stage procedure to obtain the optimal keep rate matrix and weight matrix to realize the whole training model. Extensive experiments have been conducted to demonstrate the effectiveness of LocalDrop in different models by comparing it with several algorithms and the effects of different hyperparameters on the final performances.