# Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

arXiv: https://arxiv.org/abs/1702.03275 · HTML: https://mlsanity.com/paper/1702.03275/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2017.

## Built on by (most cited first, 30 of 128)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Selective Structured State-Spaces for Long-Form Video Understanding](https://mlsanity.com/paper/2303.14526/) (2023)
- [DELTA: degradation-free fully test-time adaptation](https://mlsanity.com/paper/2301.13018/) (2023)
- [Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2407.07532/) (2024)
- [SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG](https://mlsanity.com/paper/2206.01323/) (2022)
- [MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth](https://mlsanity.com/paper/2208.02541/) (2022)
- [Rethinking Text Line Recognition Models](https://mlsanity.com/paper/2104.07787/) (2021)
- [Rethinking "Batch" in BatchNorm](https://mlsanity.com/paper/2105.07576/) (2021)
- [In Search of Lost Online Test-time Adaptation: A Survey](https://mlsanity.com/paper/2310.20199/) (2023)
- [Continual Normalization: Rethinking Batch Normalization for Online Continual Learning](https://mlsanity.com/paper/2203.16102/) (2022)
- [GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation](https://mlsanity.com/paper/2207.09763/) (2022)
- [Survey on Unsupervised Domain Adaptation for Semantic Segmentation for Visual Perception in Automated Driving](https://mlsanity.com/paper/2304.11928/) (2023)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning](https://mlsanity.com/paper/2106.05956/) (2021)
- [On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis](https://mlsanity.com/paper/2203.07976/) (2022)
- [Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning](https://mlsanity.com/paper/2201.12559/) (2022)
- [New Insights for the Stability-Plasticity Dilemma in Online Continual Learning](https://mlsanity.com/paper/2302.08741/) (2023)
- [UniNeXt: Exploring A Unified Architecture for Vision Recognition](https://mlsanity.com/paper/2304.13700/) (2023)
- [Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns](https://mlsanity.com/paper/2503.03660/) (2025)
- [Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation](https://mlsanity.com/paper/2106.15287/) (2021)
- [Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning](https://mlsanity.com/paper/2101.07525/) (2021)
- [Magnitude Invariant Parametrizations Improve Hypernetwork Learning](https://mlsanity.com/paper/2304.07645/) (2023)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [OIMNet++: Prototypical Normalization and Localization-aware Learning for Person Search](https://mlsanity.com/paper/2207.10320/) (2022)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)

## Abstract (arXiv)

Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.