# New Interpretations of Normalization Methods in Deep Learning

arXiv: https://arxiv.org/abs/2006.09104 · HTML: https://mlsanity.com/paper/2006.09104/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [Rethinking Normalization Methods in Federated Learning](https://mlsanity.com/paper/2210.03277/) (2022)
- [Normalization Enhances Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2306.00656/) (2023)
- [Preference Learning with Lie Detectors can Induce Honesty or Evasion](https://mlsanity.com/paper/2505.13787/) (2025)
- [The Brownian motion in the transformer model](https://mlsanity.com/paper/2107.05264/) (2021)
- [TSGB: Target-Selective Gradient Backprop for Probing CNN Visual Saliency](https://mlsanity.com/paper/2110.05182/) (2021)
- [Out-of-distribution Few-shot Learning For Edge Devices without Model Fine-tuning](https://mlsanity.com/paper/2304.06309/) (2023)
- [Skeleton-Guided Sparse Anchors for Rotated Instance Segmentation in Cell Microscopy](https://mlsanity.com/paper/2401.09895/) (2024)
- [An Integrated Deep Learning Model for Skin Cancer Detection Using Hybrid Feature Fusion Technique](https://mlsanity.com/paper/2410.14489/) (2024)

## Abstract (arXiv)

In recent years, a variety of normalization methods have been proposed to help train neural networks, such as batch normalization (BN), layer normalization (LN), weight normalization (WN), group normalization (GN), etc. However, mathematical tools to analyze all these normalization methods are lacking. In this paper, we first propose a lemma to define some necessary tools. Then, we use these tools to make a deep analysis on popular normalization methods and obtain the following conclusions: 1) Most of the normalization methods can be interpreted in a unified framework, namely normalizing pre-activations or weights onto a sphere; 2) Since most of the existing normalization methods are scaling invariant, we can conduct optimization on a sphere with scaling symmetry removed, which can help stabilize the training of network; 3) We prove that training with these normalization methods can make the norm of weights increase, which could cause adversarial vulnerability as it amplifies the attack. Finally, a series of experiments are conducted to verify these claims.