mlsanityData to 7 Oct 2026

Delving into the Estimation Shift of Batch Normalization in a Network

arXiv 2203.10778 ↗First posted 21 Mar 2022cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Batch normalization (BN) is a milestone technique in deep learning. It normalizes the activation using mini-batch statistics during training but the estimated population statistics during inference. This paper focuses on investigating the estimation of population statistics. We define the estimation shift magnitude of BN to quantitatively measure the difference between its estimated population statistics and expected ones. Our primary observation is that the estimation shift can be accumulated due to the stack of BN in a network, which has detriment effects for the test performance. We further find a batch-free normalization (BFN) can block such an accumulation of estimation shift. These observations motivate our design of XBNBlock that replace one BN with BFN in the bottleneck block of residual-style networks. Experiments on the ImageNet and COCO benchmarks show that XBNBlock consistently improves the performance of different architectures, including ResNet and ResNeXt, by a significant margin and seems to be more robust to distribution shift.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

51 of 59 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2017Mask R-CNNcited by 7,259
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2016Layer Normalizationcited by 5,579
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
38 more
  1. 2016Wide Residual Networkscited by 3,057
  2. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  3. 2018Group Normalizationcited by 1,514
  4. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  7. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  8. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  9. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  10. 2018Two at Once: Enhancing Learning and Generalization Capacities via IBN-Netcited by 361
  11. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  12. 2018How Does Batch Normalization Help Optimization?cited by 355
  13. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  14. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 235
  15. 2019U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translationcited by 186
  16. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 160
  17. 2018Understanding Batch Normalizationcited by 157
  18. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 127
  19. 2019Switchable Whitening for Deep Representation Learningcited by 105
  20. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  21. 2018Decorrelated Batch Normalizationcited by 83
  22. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  23. 2018Batch-Instance Normalization for Adaptively Style-Invariant Neural Networkscited by 77
  24. 2020Revisiting Batch Normalization for Improving Corruption Robustnesscited by 55
  25. 2020TaskNorm: Rethinking Batch Normalization for Meta-Learningcited by 50
  26. 2015Natural Neural Networkscited by 44
  27. 2021Rethinking "Batch" in BatchNormcited by 44
  28. 2018Differentiable Learning-to-Normalize via Switchable Normalizationcited by 33
  29. 2019Online Normalization for Training Neural Networkscited by 29
  30. 2019Four Things Everyone Should Know to Improve Batch Normalizationcited by 28
  31. 2019Positional Normalizationcited by 28
  32. 2019EvalNorm: Estimating Batch Normalization Statistics for Evaluationcited by 20
  33. 2020Cross-Iteration Batch Normalizationcited by 20
  34. 2020Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalizationcited by 19
  35. 2019SSN: Learning Sparse Switchable Normalization via SparsestMaxcited by 14
  36. 2020An Investigation into the Stochasticity of Batch Whiteningcited by 14
  37. 2020Group Whitening: Balancing Learning Efficiency and Representational Capacitycited by 13
  38. 2019Unsupervised Out-of-Distribution Detection with Batch Normalizationcited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 1; 2024: 1; 2025: 1.

7 papers.

  1. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  2. 2022L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Featurescited by 0
  3. 2022Adaptive Batch Normalization for Training Data with Heterogeneous Featurescited by 0
  4. 2022Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learningcited by 0
  5. 2023Patch-aware Batch Normalization for Improving Cross-domain Robustnesscited by 0
  6. 2024DATTA: Domain Diversity Aware Test-Time Adaptation for Dynamic Domain Shift Data Streamscited by 0
  7. 2025An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithmscited by 0