# Cross-Iteration Batch Normalization

arXiv: https://arxiv.org/abs/2002.05712 · HTML: https://mlsanity.com/paper/2002.05712/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 20 of 20)

- [Rethinking "Batch" in BatchNorm](https://mlsanity.com/paper/2105.07576/) (2021)
- [What is YOLOv5: A deep look into the internal features of the popular object detector](https://mlsanity.com/paper/2407.20892/) (2024)
- [Towards Online Domain Adaptive Object Detection](https://mlsanity.com/paper/2204.05289/) (2022)
- [Your Group-Relative Advantage Is Biased](https://mlsanity.com/paper/2601.08521/) (2026)
- [On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis](https://mlsanity.com/paper/2203.07976/) (2022)
- [YOLOv1 to YOLOv10: A comprehensive review of YOLO variants and their application in the agricultural domain](https://mlsanity.com/paper/2406.10139/) (2024)
- [OIMNet++: Prototypical Normalization and Localization-aware Learning for Person Search](https://mlsanity.com/paper/2207.10320/) (2022)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)
- [A fast accurate fine-grain object detection model based on YOLOv4 deep neural network](https://mlsanity.com/paper/2111.00298/) (2021)
- [Logit Normalization for Long-tail Object Detection](https://mlsanity.com/paper/2203.17020/) (2022)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Revisiting LARS for Large Batch Training Generalization of Neural Networks](https://mlsanity.com/paper/2309.14053/) (2023)
- [Probabilistic Approach for Road-Users Detection](https://mlsanity.com/paper/2112.01360/) (2021)
- [Separable Batch Normalization for Robust Facial Landmark Localization with Cross-protocol Network Training](https://mlsanity.com/paper/2101.06663/) (2021)
- [Scale Equalization for Multi-Level Feature Fusion](https://mlsanity.com/paper/2402.01149/) (2024)
- [L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features](https://mlsanity.com/paper/2207.02625/) (2022)
- [Adaptive Batch Normalization for Training Data with Heterogeneous Features](https://mlsanity.com/paper/2211.02050/) (2022)
- [Adaptive Cross Batch Normalization for Metric Learning](https://mlsanity.com/paper/2303.17127/) (2023)
- [Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework](https://mlsanity.com/paper/2311.15993/) (2023)
- [Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning](https://mlsanity.com/paper/2505.21877/) (2025)

## Abstract (arXiv)

A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique. Code is available at https://github.com/Howal/Cross-iterationBatchNorm .