mlsanityData to 14 Oct 2026

Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

arXiv 1702.03275 ↗First posted 10 Feb 2017cs.LGcited by 128

Not on the map. Why?

Abstract (arXiv)

Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 34; 2023: 35; 2024: 14; 2025: 14; 2026: 7.

128 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,579
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  4. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  5. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  6. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  7. 2023DELTA: degradation-free fully test-time adaptationcited by 72
  8. 2024Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimationcited by 59
  9. 2022SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEGcited by 53
  10. 2022MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depthcited by 53
  11. 2021Rethinking Text Line Recognition Modelscited by 45
  12. 2021Rethinking "Batch" in BatchNormcited by 45
38 more
  1. 2023In Search of Lost Online Test-time Adaptation: A Surveycited by 44
  2. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  3. 2022GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentationcited by 35
  4. 2023Survey on Unsupervised Domain Adaptation for Semantic Segmentation for Visual Perception in Automated Drivingcited by 29
  5. 2023Semantic Image Segmentation: Two Decades of Researchcited by 27
  6. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  7. 2022On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysiscited by 19
  8. 2022Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learningcited by 16
  9. 2023New Insights for the Stability-Plasticity Dilemma in Online Continual Learningcited by 16
  10. 2023UniNeXt: Exploring A Unified Architecture for Vision Recognitioncited by 16
  11. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  12. 2021Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentationcited by 15
  13. 2021Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learningcited by 14
  14. 2023Magnitude Invariant Parametrizations Improve Hypernetwork Learningcited by 14
  15. 2022Deep transfer learning for image classification: a surveycited by 13
  16. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  17. 2022OIMNet++: Prototypical Normalization and Localization-aware Learning for Person Searchcited by 12
  18. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  19. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  20. 2023Shoggoth: Towards Efficient Edge-Cloud Collaborative Real-Time Video Inference via Adaptive Online Learningcited by 11
  21. 2024Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillationcited by 11
  22. 2021Deep Learning for Scene Classification: A Surveycited by 9
  23. 2021Continual Learning on the Edge with TensorFlow Litecited by 9
  24. 2021Stochastic Whitening Batch Normalizationcited by 9
  25. 2023Incremental Learning for Heterogeneous Structure Segmentation in Brain Tumor MRIcited by 9
  26. 2022AtmoDist: Self-supervised Representation Learning for Atmospheric Dynamicscited by 8
  27. 2022Alleviating Representational Shift for Continual Fine-tuningcited by 8
  28. 2022Batch Normalization Explainedcited by 8
  29. 2023Experimenting with Normalization Layers in Federated Learning on non-IID scenarioscited by 8
  30. 2023Bag of Tricks for Fully Test-Time Adaptationcited by 8
  31. 2023GenQ: Quantization in Low Data Regimes with Generative Synthetic Datacited by 8
  32. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?cited by 8
  33. 2021Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneitycited by 7
  34. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  35. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  36. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  37. 2022Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognitioncited by 6
  38. 2023Dual Residual Attention Network for Image Denoisingcited by 6

50 most cited of 128 All 128 (JSON)