# Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift

arXiv: https://arxiv.org/abs/1801.05134 · HTML: https://mlsanity.com/paper/1801.05134/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2018.

## Built on by (most cited first, 30 of 37)

- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Rethinking "Batch" in BatchNorm](https://mlsanity.com/paper/2105.07576/) (2021)
- [DualNet: Continual Learning, Fast and Slow](https://mlsanity.com/paper/2110.00175/) (2021)
- [SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization](https://mlsanity.com/paper/2405.11582/) (2024)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)
- [Design Smells in Deep Learning Programs: An Empirical Study](https://mlsanity.com/paper/2107.02279/) (2021)
- [Where Do Deep Fakes Look? Synthetic Face Detection via Gaze Tracking](https://mlsanity.com/paper/2101.01165/) (2021)
- [Automatic Fault Detection for Deep Learning Programs Using Graph Transformations](https://mlsanity.com/paper/2105.08095/) (2021)
- [How to Use Dropout Correctly on Residual Networks with Batch Normalization](https://mlsanity.com/paper/2302.06112/) (2023)
- [Multirate Training of Neural Networks](https://mlsanity.com/paper/2106.10771/) (2021)
- [Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networks](https://mlsanity.com/paper/2106.00673/) (2021)
- [Batch Normalization Is Blind to the First and Second Derivatives of the Loss](https://mlsanity.com/paper/2205.15146/) (2022)
- [DiffLoad: Uncertainty Quantification in Electrical Load Forecasting with the Diffusion Model](https://mlsanity.com/paper/2306.01001/) (2023)
- [Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models](https://mlsanity.com/paper/2402.16255/) (2024)
- [Adaptive Dropout: Unleashing Dropout across Layers for Generalizable Image Super-Resolution](https://mlsanity.com/paper/2506.12738/) (2025)
- [Testing Feedforward Neural Networks Training Programs](https://mlsanity.com/paper/2204.00694/) (2022)
- [Functional Network: A Novel Framework for Interpretability of Deep Neural Networks](https://mlsanity.com/paper/2205.11702/) (2022)
- [Efficient brain age prediction from 3D MRI volumes using 2D projections](https://mlsanity.com/paper/2211.05762/) (2022)
- [Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics](https://mlsanity.com/paper/2212.10430/) (2022)
- [Dropout Drops Double Descent](https://mlsanity.com/paper/2305.16179/) (2023)
- [Interpretable Convolutional SyncNet](https://mlsanity.com/paper/2409.00971/) (2024)
- [Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction](https://mlsanity.com/paper/2206.00913/) (2022)
- [Measuring Overfitting in Convolutional Neural Networks using Adversarial Perturbations and Label Noise](https://mlsanity.com/paper/2209.13382/) (2022)
- [Cosmic Microwave Background Recovery: A Graph-Based Bayesian Convolutional Network Approach](https://mlsanity.com/paper/2302.12378/) (2023)
- [Weight Compander: A Simple Weight Reparameterization for Regularization](https://mlsanity.com/paper/2306.16993/) (2023)
- [Out-of-distribution Object Detection through Bayesian Uncertainty Estimation](https://mlsanity.com/paper/2310.19119/) (2023)
- [Continual Learning, Fast and Slow](https://mlsanity.com/paper/2209.02370/) (2022)
- [Predicting Mutual Funds' Performance using Deep Learning and Ensemble Techniques](https://mlsanity.com/paper/2209.09649/) (2022)
- [Uncertainty estimation for out-of-distribution detection in computational histopathology](https://mlsanity.com/paper/2210.09909/) (2022)
- [Robust Millimeter Beamforming via Self-Supervised Hybrid Deep Learning](https://mlsanity.com/paper/2303.12653/) (2023)

## Abstract (arXiv)

This paper first answers the question "why do the two most powerful techniques Dropout and Batch Normalization (BN) often lead to a worse performance when they are combined together?" in both theoretical and statistical aspects. Theoretically, we find that Dropout would shift the variance of a specific neural unit when we transfer the state of that network from train to test. However, BN would maintain its statistical variance, which is accumulated from the entire learning procedure, in the test phase. The inconsistency of that variance (we name this scheme as "variance shift") causes the unstable numerical behavior in inference that leads to more erroneous predictions finally, when applying Dropout before BN. Thorough experiments on DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to the uncovered mechanism, we next explore several strategies that modifies Dropout and try to overcome the limitations of their combination by avoiding the variance shift risks.