# Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

arXiv: https://arxiv.org/abs/1611.04520 · HTML: https://mlsanity.com/paper/1611.04520/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2016.

## Built on by (most cited first, 12 of 12)

- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [Neural Sampling Machine with Stochastic Synapse allows Brain-like Learning and Inference](https://mlsanity.com/paper/2102.10477/) (2021)
- [Unconditional stability of a recurrent neural circuit implementing divisive normalization](https://mlsanity.com/paper/2409.18946/) (2024)
- [Analytically Tractable Inference in Deep Neural Networks](https://mlsanity.com/paper/2103.05461/) (2021)
- [Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations](https://mlsanity.com/paper/2202.13074/) (2022)
- [Kernel Normalized Convolutional Networks](https://mlsanity.com/paper/2205.10089/) (2022)
- [Neural Networks with Divisive normalization for image segmentation with application in cityscapes dataset](https://mlsanity.com/paper/2203.13558/) (2022)
- [Theoretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Rate](https://mlsanity.com/paper/2209.07587/) (2022)
- [Enhancing Neural Network Representations with Prior Knowledge-Based Normalization](https://mlsanity.com/paper/2403.16798/) (2024)
- [Parametric PerceptNet: A bio-inspired deep-net trained for Image Quality Assessment](https://mlsanity.com/paper/2412.03210/) (2024)
- [HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification](https://mlsanity.com/paper/2610.11534/) (2026)

## Abstract (arXiv)

Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural networks. On the other hand, layer normalization normalizes the activations across all activities within a layer. This was shown to work well in the recurrent setting. In this paper we propose a unified view of normalization techniques, as forms of divisive normalization, which includes layer and batch normalization as special cases. Our second contribution is the finding that a small modification to these normalization schemes, in conjunction with a sparse regularizer on the activations, leads to significant benefits over standard normalization techniques. We demonstrate the effectiveness of our unified divisive normalization framework in the context of convolutional neural nets and recurrent neural networks, showing improvements over baselines in image classification, language modeling as well as super-resolution.