L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features

In this paper, we analyze batch normalization from the perspective of discriminability and find the disadvantages ignored by previous studies: the difference in l2 norms of sample features can hinder batch normalization from obtaining more distinguished inter-class features and more compact intra-class features. To address this issue, we propose a simple yet effective method to equalize the l2 norms of sample features. Concretely, we l2-normalize each sample feature before feeding them into batch normalization, and therefore the features are of the same magnitude. Since the proposed method combines the l2 normalization and batch normalization, we name our method L2BN. The L2BN can strengthen the compactness of intra-class features and enlarge the discrepancy of inter-class features. The L2BN is easy to implement and can exert its effect without any additional parameters or hyper-parameters. We evaluate the effectiveness of L2BN through extensive experiments with various models on image classification and acoustic scene classification tasks. The results demonstrate that the L2BN can boost the generalization ability of various neural network models and achieve considerable performance improvements.