mlsanityData to 7 Oct 2026

Rethinking "Batch" in BatchNorm

arXiv 2105.07576 ↗First posted 17 May 2021cs.CVcited by 44

Not on the map. Why?

Abstract (arXiv)

BatchNorm is a critical building block in modern convolutional neural networks. Its unique property of operating on "batches" instead of individual samples introduces significantly different behaviors from most other operations in deep learning. As a result, it leads to many hidden caveats that can negatively impact model's performance in subtle ways. This paper thoroughly reviews such problems in visual recognition tasks, and shows that a key to address them is to rethink different choices in the concept of "batch" in BatchNorm. By presenting these caveats and their mitigations, we hope this review can help researchers use BatchNorm more effectively.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 67 of its 85 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

67 of 85 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2017Mask R-CNNcited by 7,259
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2016Layer Normalizationcited by 5,579
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2015Fast R-CNNcited by 2,930
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2018Group Normalizationcited by 1,514
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  12. 2017Mixed Precision Trainingcited by 1,135
  13. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  14. 2016Deep Networks with Stochastic Depthcited by 973
  15. 2020Designing Network Design Spacescited by 954
  16. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  17. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  18. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  19. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  20. 2018Rethinking ImageNet Pre-trainingcited by 376
  21. 2018Two at Once: Enhancing Learning and Generalization Capacities via IBN-Netcited by 361
  22. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  23. 2018Image Generation from Scene Graphscited by 324
  24. 2020A Metric Learning Reality Checkcited by 317
  25. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  26. 2018Context Encoding for Semantic Segmentationcited by 289
  27. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  28. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  29. 2019Adversarial Examples Improve Image Recognitioncited by 283
  30. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  31. 2018DropBlock: A regularization method for convolutional networkscited by 261
  32. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 235
  33. 2019Domain-Specific Batch Normalization for Unsupervised Domain Adaptationcited by 197
  34. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 127
  35. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  36. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78
  37. 2017AutoDIAL: Automatic DomaIn Alignment Layerscited by 75
  38. 2020On Feature Normalization and Data Augmentationcited by 71

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 17; 2023: 8; 2024: 4; 2025: 4; 2026: 4.

44 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  3. 2024The Road Less Scheduledcited by 158
  4. 20223D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentationcited by 94
  5. 2022DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detectioncited by 48
  6. 2022Video Test-Time Adaptation for Action Recognitioncited by 35
  7. 2021Learning from Temporal Gradient for Semi-supervised Action Recognitioncited by 26
  8. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  9. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  10. 2022On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysiscited by 19
  11. 2022RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalizationcited by 19
  12. 2022Learning Instance-Specific Adaptation for Cross-Domain Segmentationcited by 13
32 more
  1. 2024CTBENCH: A Library and Benchmark for Certified Trainingcited by 13
  2. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  3. 2022Rethinking Performance Gains in Image Dehazing Networkscited by 12
  4. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  5. 2023Differentially Private Video Activity Recognitioncited by 9
  6. 2022Normalization Perturbation: A Simple Domain Generalization Method for Real-World Domain Shiftscited by 8
  7. 2021Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating roomcited by 7
  8. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  9. 2022Modular Degradation Simulation and Restoration for Under-Display Cameracited by 7
  10. 2023INoD: Injected Noise Discriminator for Self-Supervised Representation Learning in Agricultural Fieldscited by 6
  11. 2023Benchmarking White Blood Cell Classification Under Domain Shiftcited by 4
  12. 2022Generalization to translation shifts: a study in architectures and augmentationscited by 3
  13. 2023On the Training Instability of Shuffling SGD with Batch Normalizationcited by 3
  14. 2024QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devicescited by 3
  15. 2025Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularizationcited by 3
  16. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 3
  17. 2021Training BatchNorm Only in Neural Architecture Search and Beyondcited by 2
  18. 2022NoMorelization: Building Normalizer-Free Models from a Sample's Perspectivecited by 2
  19. 2022LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Imagescited by 2
  20. 2023Ghost Noise for Regularizing Deep Neural Networkscited by 1
  21. 2024Fortifying Fully Convolutional Generative Adversarial Networks for Image Super-Resolution Using Divergence Measurescited by 1
  22. 2021New Perspective on Progressive GANs Distillation for One-class Novelty Detectioncited by 0
  23. 2022L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Featurescited by 0
  24. 2022A simple normalization technique using window statistics to improve the out-of-distribution generalization on medical imagescited by 0
  25. 2023Out-of-distribution Few-shot Learning For Edge Devices without Model Fine-tuningcited by 0
  26. 2023The Counterattack of CNNs in Self-Supervised Learning: Larger Kernel Size might be All You Needcited by 0
  27. 2025SC3EF: A Joint Self-Correlation and Cross-Correspondence Estimation Framework for Visible and Thermal Image Registrationcited by 0
  28. 2025Spooky Action at a Distance: Normalization Layers Enable Side-Channel Spatial Communicationcited by 0
  29. 2025Self Identity Mappingcited by 0
  30. 2026Beyond receptive fields: sequence-pooled normalization can supply most of a sequence labeler's contextcited by 0
  31. 2026Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpointscited by 0
  32. 2026Input-Layer Starvation: Why Per-Layer Pruning Breaks IoT Intrusion Detectorscited by 0