# Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients

arXiv: https://arxiv.org/abs/1907.09539 · HTML: https://mlsanity.com/paper/1907.09539/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2019.

## Built on by (most cited first, 3 of 3)

- [Differentially private federated deep learning for multi-site medical image segmentation](https://mlsanity.com/paper/2107.02586/) (2021)
- [AdaStereo: An Efficient Domain-Adaptive Stereo Matching Approach](https://mlsanity.com/paper/2112.04974/) (2021)
- [Evaluating Online and Offline Accuracy Traversal Algorithms for k-Complete Neural Network Architectures](https://mlsanity.com/paper/2101.06518/) (2021)

## Abstract (arXiv)

Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior or the deep decoder. We show that for this setup, channel normalization, which centers and normalizes each channel individually, avoids vanishing gradients, whereas, without normalization, gradients vanish which prevents efficient optimization. This effect prevails in deep single-channel linear convolutional networks, and we show that without channel normalization, gradient descent takes at least exponentially many steps to come close to an optimum. Contrary, with channel normalization, the gradients remain bounded, thus avoiding exploding gradients.