# Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

arXiv: https://arxiv.org/abs/1610.06160 · HTML: https://mlsanity.com/paper/1610.06160/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2016.

## Built on by (most cited first, 4 of 4)

- [Stochastic Whitening Batch Normalization](https://mlsanity.com/paper/2106.04413/) (2021)
- [Batch Layer Normalization, A new normalization layer for CNNs and RNN](https://mlsanity.com/paper/2209.08898/) (2022)
- [IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning](https://mlsanity.com/paper/2510.25262/) (2025)
- [BCN: Batch Channel Normalization for Image Classification](https://mlsanity.com/paper/2312.00596/) (2023)

## Abstract (arXiv)

We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed --- recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations.