mlsanityData to 7 Oct 2026

Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift

arXiv 1801.05134 ↗First posted 16 Jan 2018cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

This paper first answers the question "why do the two most powerful techniques Dropout and Batch Normalization (BN) often lead to a worse performance when they are combined together?" in both theoretical and statistical aspects. Theoretically, we find that Dropout would shift the variance of a specific neural unit when we transfer the state of that network from train to test. However, BN would maintain its statistical variance, which is accumulated from the entire learning procedure, in the test phase. The inconsistency of that variance (we name this scheme as "variance shift") causes the unstable numerical behavior in inference that leads to more erroneous predictions finally, when applying Dropout before BN. Thorough experiments on DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to the uncovered mechanism, we next explore several strategies that modifies Dropout and try to overcome the limitations of their combination by avoiding the variance shift risks.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 10; 2024: 6; 2025: 1; 2026: 1.

36 papers.

Sort:

By area:

  1. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  2. 2021Rethinking "Batch" in BatchNormcited by 44
  3. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  4. 2024SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalizationcited by 17
  5. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  6. 2021Design Smells in Deep Learning Programs: An Empirical Studycited by 10
  7. 2021Where Do Deep Fakes Look? Synthetic Face Detection via Gaze Trackingcited by 9
  8. 2021Automatic Fault Detection for Deep Learning Programs Using Graph Transformationscited by 9
  9. 2023How to Use Dropout Correctly on Residual Networks with Batch Normalizationcited by 7
  10. 2021Multirate Training of Neural Networkscited by 5
  11. 2021Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networkscited by 3
  12. 2022Batch Normalization Is Blind to the First and Second Derivatives of the Losscited by 3
24 more
  1. 2023DiffLoad: Uncertainty Quantification in Electrical Load Forecasting with the Diffusion Modelcited by 3
  2. 2024Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Modelscited by 3
  3. 2025Adaptive Dropout: Unleashing Dropout across Layers for Generalizable Image Super-Resolutioncited by 3
  4. 2022Testing Feedforward Neural Networks Training Programscited by 2
  5. 2022Functional Network: A Novel Framework for Interpretability of Deep Neural Networkscited by 2
  6. 2022Efficient brain age prediction from 3D MRI volumes using 2D projectionscited by 2
  7. 2022Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamicscited by 2
  8. 2023Dropout Drops Double Descentcited by 2
  9. 2022Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reductioncited by 1
  10. 2022Measuring Overfitting in Convolutional Neural Networks using Adversarial Perturbations and Label Noisecited by 1
  11. 2023Cosmic Microwave Background Recovery: A Graph-Based Bayesian Convolutional Network Approachcited by 1
  12. 2023Weight Compander: A Simple Weight Reparameterization for Regularizationcited by 1
  13. 2023Out-of-distribution Object Detection through Bayesian Uncertainty Estimationcited by 1
  14. 2024Interpretable Convolutional SyncNetcited by 1
  15. 2022Continual Learning, Fast and Slowcited by 0
  16. 2022Predicting Mutual Funds' Performance using Deep Learning and Ensemble Techniquescited by 0
  17. 2022Uncertainty estimation for out-of-distribution detection in computational histopathologycited by 0
  18. 2023Robust Millimeter Beamforming via Self-Supervised Hybrid Deep Learningcited by 0
  19. 2023Regularization Through Simultaneous Learning: A Case Study on Plant Classificationcited by 0
  20. 2023Combining transmission speckle photography and convolutional neural network for determination of fat content in cow's milk -- an exercise in classification of parameters of a complex suspensioncited by 0
  21. 2023Mesh-Wise Prediction of Demographic Composition from Satellite Images Using Multi-Head Convolutional Neural Networkcited by 0
  22. 2024Stochastic Subsampling With Average Poolingcited by 0
  23. 2024DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learningcited by 0
  24. 2026Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Modelscited by 0