# L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features

arXiv: https://arxiv.org/abs/2207.02625 · HTML: https://mlsanity.com/paper/2207.02625/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 43 of its 54 references that are in the map.
- First posted on arXiv 6 Jul 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [Norm matters: efficient and accurate normalization schemes in deep networks](https://mlsanity.com/paper/1803.01814/) (2018)
- [Deep Hyperspherical Learning](https://mlsanity.com/paper/1711.03189/) (2017)
- [Micro-Batch Training with Batch-Channel Normalization and Weight Standardization](https://mlsanity.com/paper/1903.10520/) (2019)
- [Decoupled Networks](https://mlsanity.com/paper/1804.08071/) (2018)
- [Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks](https://mlsanity.com/paper/1911.09737/) (2019)

## Abstract (arXiv)

In this paper, we analyze batch normalization from the perspective of discriminability and find the disadvantages ignored by previous studies: the difference in l2 norms of sample features can hinder batch normalization from obtaining more distinguished inter-class features and more compact intra-class features. To address this issue, we propose a simple yet effective method to equalize the l2 norms of sample features. Concretely, we l2-normalize each sample feature before feeding them into batch normalization, and therefore the features are of the same magnitude. Since the proposed method combines the l2 normalization and batch normalization, we name our method L2BN. The L2BN can strengthen the compactness of intra-class features and enlarge the discrepancy of inter-class features. The L2BN is easy to implement and can exert its effect without any additional parameters or hyper-parameters. We evaluate the effectiveness of L2BN through extensive experiments with various models on image classification and acoustic scene classification tasks. The results demonstrate that the L2BN can boost the generalization ability of various neural network models and achieve considerable performance improvements.