# Scale Normalization

arXiv: https://arxiv.org/abs/1604.07796 · HTML: https://mlsanity.com/paper/1604.07796/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2016.

## Built on by (most cited first, 1 of 1)

- [KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmark](https://mlsanity.com/paper/2609.19840/) (2026)

## Abstract (arXiv)

One of the difficulties of training deep neural networks is caused by improper scaling between layers. Scaling issues introduce exploding / gradient problems, and have typically been addressed by careful scale-preserving initialization. We investigate the value of preserving scale, or isometry, beyond the initial weights. We propose two methods of maintaing isometry, one exact and one stochastic. Preliminary experiments show that for both determinant and scale-normalization effectively speeds up learning. Results suggest that isometry is important in the beginning of learning, and maintaining it leads to faster learning.