# The Unreasonable Effectiveness of the Final Batch Normalization Layer

arXiv: https://arxiv.org/abs/2109.09016 · HTML: https://mlsanity.com/paper/2109.09016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Using Deep Learning for Image-Based Plant Disease Detection](https://mlsanity.com/paper/1604.03169/) (2016)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Deep Learning on Small Datasets without Pre-Training using Cosine Loss](https://mlsanity.com/paper/1901.09054/) (2019)
- [Robust Anomaly Detection in Images using Adversarial Autoencoders](https://mlsanity.com/paper/1901.06355/) (2019)
- [Adaptive Estimators Show Information Compression in Deep Neural Networks](https://mlsanity.com/paper/1902.09037/) (2019)
- [Improving Model Accuracy for Imbalanced Image Classification Tasks by Adding a Final Batch Normalization Layer: An Empirical Study](https://mlsanity.com/paper/2011.06319/) (2020)

## Abstract (arXiv)

Early-stage disease indications are rarely recorded in real-world domains, such as Agriculture and Healthcare, and yet, their accurate identification is critical in that point of time. In this type of highly imbalanced classification problems, which encompass complex features, deep learning (DL) is much needed because of its strong detection capabilities. At the same time, DL is observed in practice to favor majority over minority classes and consequently suffer from inaccurate detection of the targeted early-stage indications. In this work, we extend the study done by Kocaman et al., 2020, showing that the final BN layer, when placed before the softmax output layer, has a considerable impact in highly imbalanced image classification problems as well as undermines the role of the softmax outputs as an uncertainty measure. This current study addresses additional hypotheses and reports on the following findings: (i) the performance gain after adding the final BN layer in highly imbalanced settings could still be achieved after removing this additional BN layer in inference; (ii) there is a certain threshold for the imbalance ratio upon which the progress gained by the final BN layer reaches its peak; (iii) the batch size also plays a role and affects the outcome of the final BN application; (iv) the impact of the BN application is also reproducible on other datasets and when utilizing much simpler neural architectures; (v) the reported BN effect occurs only per a single majority class and multiple minority classes i.e., no improvements are evident when there are two majority classes; and finally, (vi) utilizing this BN layer with sigmoid activation has almost no impact when dealing with a strongly imbalanced image classification tasks.