# Effective and Efficient Dropout for Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1904.03392 · HTML: https://mlsanity.com/paper/1904.03392/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 21 of 21)

- [Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results](https://mlsanity.com/paper/2204.03475/) (2022)
- [How to Use Dropout Correctly on Residual Networks with Batch Normalization](https://mlsanity.com/paper/2302.06112/) (2023)
- [Decoding ECoG signal into 3D hand translation using deep learning](https://mlsanity.com/paper/2110.03528/) (2021)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Noise Injection-based Regularization for Point Cloud Processing](https://mlsanity.com/paper/2103.15027/) (2021)
- [Revisiting Structured Dropout](https://mlsanity.com/paper/2210.02570/) (2022)
- [Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling](https://mlsanity.com/paper/2303.15973/) (2023)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Evaluating CNN with Oscillatory Activation Function](https://mlsanity.com/paper/2211.06878/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)
- [R-Block: Regularized Block of Dropout for convolutional networks](https://mlsanity.com/paper/2307.15150/) (2023)
- [Robust-MBDL: A Robust Multi-branch Deep Learning Based Model for Remaining Useful Life Prediction and Operational Condition Identification of Rotating Machines](https://mlsanity.com/paper/2309.06157/) (2023)
- [Brain Tumor Segmentation with Special Emphasis on the Non-Enhancing Brain Tumor Compartment](https://mlsanity.com/paper/2602.21703/) (2026)
- [Explicit Dropout: Deterministic Regularization for Transformer Architectures](https://mlsanity.com/paper/2604.20505/) (2026)

## Abstract (arXiv)

Convolutional Neural networks (CNNs) based applications have become ubiquitous, where proper regularization is greatly needed. To prevent large neural network models from overfitting, dropout has been widely used as an efficient regularization technique in practice. However, many recent works show that the standard dropout is ineffective or even detrimental to the training of CNNs. In this paper, we revisit this issue and examine various dropout variants in an attempt to improve existing dropout-based regularization techniques for CNNs. We attribute the failure of standard dropout to the conflict between the stochasticity of dropout and its following Batch Normalization (BN), and propose to reduce the conflict by placing dropout operations right before the convolutional operation instead of BN, or totally address this issue by replacing BN with Group Normalization (GN). We further introduce a structurally more suited dropout variant Drop-Conv2d, which provides more efficient and effective regularization for deep CNNs. These dropout variants can be readily integrated into the building blocks of CNNs and implemented in existing deep learning platforms. Extensive experiments on benchmark datasets including CIFAR, SVHN and ImageNet are conducted to compare the existing building blocks and the proposed ones with dropout training. Results show that our building blocks improve over state-of-the-art CNNs significantly, which is mainly due to the better regularization and implicit model ensemble effect.