# Understanding Weight Normalized Deep Neural Networks with Rectified Linear Units

arXiv: https://arxiv.org/abs/1810.01877 · HTML: https://mlsanity.com/paper/1810.01877/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2018.

## Built on by (most cited first, 2 of 2)

- [Hidden Synergy: L1 Weight Normalization and 1-Path-Norm Regularization](https://mlsanity.com/paper/2404.19112/) (2024)
- [A Statistician Teaches Deep Learning](https://mlsanity.com/paper/2102.01194/) (2021)

## Abstract (arXiv)

This paper presents a general framework for norm-based capacity control for Lp,q weight normalized deep neural networks. We establish the upper bound on the Rademacher complexities of this family. With an Lp,q normalization where q≤ p^*, and 1/p+1/p*=1, we discuss properties of a width-independent capacity control, which only depends on depth by a square root term. We further analyze the approximation properties of Lp,q weight normalized deep neural networks. In particular, for an L1,∞ weight normalized network, the approximation error can be controlled by the L1 norm of the output layer, and the corresponding generalization error only depends on the architecture by the square root of the depth.