# Extended Batch Normalization

arXiv: https://arxiv.org/abs/2003.05569 · HTML: https://mlsanity.com/paper/2003.05569/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2020.

## Built on by (most cited first, 5 of 5)

- [FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes](https://mlsanity.com/paper/2110.08059/) (2021)
- [Signature-Graph Networks](https://mlsanity.com/paper/2110.11551/) (2021)
- [GCCN: Global Context Convolutional Network](https://mlsanity.com/paper/2110.11664/) (2021)
- [BCN: Batch Channel Normalization for Image Classification](https://mlsanity.com/paper/2312.00596/) (2023)
- [HyperZ·Z·W Operator Connects Slow-Fast Networks for Full Context Interaction](https://mlsanity.com/paper/2401.17948/) (2024)

## Abstract (arXiv)

Batch normalization (BN) has become a standard technique for training the modern deep networks. However, its effectiveness diminishes when the batch size becomes smaller, since the batch statistics estimation becomes inaccurate. That hinders batch normalization's usage for 1) training larger model which requires small batches constrained by memory consumption, 2) training on mobile or embedded devices of which the memory resource is limited. In this paper, we propose a simple but effective method, called extended batch normalization (EBN). For NCHW format feature maps, extended batch normalization computes the mean along the (N, H, W) dimensions, as the same as batch normalization, to maintain the advantage of batch normalization. To alleviate the problem caused by small batch size, extended batch normalization computes the standard deviation along the (N, C, H, W) dimensions, thus enlarges the number of samples from which the standard deviation is computed. We compare extended batch normalization with batch normalization and group normalization on the datasets of MNIST, CIFAR-10/100, STL-10, and ImageNet, respectively. The experiments show that extended batch normalization alleviates the problem of batch normalization with small batch size while achieving close performances to batch normalization with large batch size.