# How to Use Dropout Correctly on Residual Networks with Batch Normalization

arXiv: https://arxiv.org/abs/2302.06112 · HTML: https://mlsanity.com/paper/2302.06112/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Revisiting ResNets: Improved Training and Scaling Strategies](https://mlsanity.com/paper/2103.07579/) (2021)
- [Online Deep Clustering for Unsupervised Representation Learning](https://mlsanity.com/paper/2006.10645/) (2020)
- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift](https://mlsanity.com/paper/1801.05134/) (2018)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [STConvS2S: Spatiotemporal Convolutional Sequence to Sequence Network for Weather Forecasting](https://mlsanity.com/paper/1912.00134/) (2019)
- [Effective and Efficient Dropout for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1904.03392/) (2019)
- [Offline Handwritten Chinese Text Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/2006.15619/) (2020)

## Built on by (most cited first, 7 of 7)

- [TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensors](https://mlsanity.com/paper/2407.21453/) (2024)
- [Adaptive Dropout: Unleashing Dropout across Layers for Generalizable Image Super-Resolution](https://mlsanity.com/paper/2506.12738/) (2025)
- [Enhanced Uncertainty Estimation in Ultrasound Image Segmentation with MSU-Net](https://mlsanity.com/paper/2407.21273/) (2024)
- [Scaling Laws for Uncertainty in Deep Learning](https://mlsanity.com/paper/2506.09648/) (2025)
- [Stochastic Subsampling With Average Pooling](https://mlsanity.com/paper/2409.16630/) (2024)
- [Domain-Agnostic Feature Modulation for Semi-Supervised Domain Generalization](https://mlsanity.com/paper/2503.20897/) (2025)
- [Uncertainty Reliability Under Domain Shift: An Investigation for Data-Driven Blood Pressure Estimation in Photoplethysmography](https://mlsanity.com/paper/2605.18008/) (2026)

## Abstract (arXiv)

For the stable optimization of deep neural networks, regularization methods such as dropout and batch normalization have been used in various tasks. Nevertheless, the correct position to apply dropout has rarely been discussed, and different positions have been employed depending on the practitioners. In this study, we investigate the correct position to apply dropout. We demonstrate that for a residual network with batch normalization, applying dropout at certain positions increases the performance, whereas applying dropout at other positions decreases the performance. Based on theoretical analysis, we provide the following guideline for the correct position to apply dropout: apply one dropout after the last batch normalization but before the last weight layer in the residual branch. We provide detailed theoretical explanations to support this claim and demonstrate them through module tests. In addition, we investigate the correct position of dropout in the head that produces the final prediction. Although the current consensus is to apply dropout after global average pooling, we prove that applying dropout before global average pooling leads to a more stable output. The proposed guidelines are validated through experiments using different datasets and models.