# On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector

arXiv: https://arxiv.org/abs/2206.13646 · HTML: https://mlsanity.com/paper/2206.13646/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't](https://mlsanity.com/paper/2009.10713/) (2020)
- [Global Convergence of Block Coordinate Descent in Deep Learning](https://mlsanity.com/paper/1803.00225/) (2018)
- [Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes](https://mlsanity.com/paper/2102.09385/) (2021)
- [Full error analysis for the training of deep neural networks](https://mlsanity.com/paper/1910.00121/) (2019)
- [How degenerate is the parametrization of neural networks with the ReLU activation function?](https://mlsanity.com/paper/1905.09803/) (2019)
- [Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Non-Convex Optimization](https://mlsanity.com/paper/1904.03537/) (2019)
- [On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks](https://mlsanity.com/paper/2112.09684/) (2021)
- [Unifying abstract inexact convergence theorems and block coordinate variable metric iPiano](https://mlsanity.com/paper/1602.07283/) (2016)
- [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)

## Abstract (arXiv)

It is an elementary fact in the scientific literature that the Lipschitz norm of the realization function of a feedforward fully-connected rectified linear unit (ReLU) artificial neural network (ANN) can, up to a multiplicative constant, be bounded from above by sums of powers of the norm of the ANN parameter vector. Roughly speaking, in this work we reveal in the case of shallow ANNs that the converse inequality is also true. More formally, we prove that the norm of the equivalence class of ANN parameter vectors with the same realization function is, up to a multiplicative constant, bounded from above by the sum of powers of the Lipschitz norm of the ANN realization function (with the exponents  1/2  and  1 ). Moreover, we prove that this upper bound only holds when employing the Lipschitz norm but does neither hold for H"older norms nor for Sobolev-Slobodeckij norms. Furthermore, we prove that this upper bound only holds for sums of powers of the Lipschitz norm with the exponents  1/2  and  1  but does not hold for the Lipschitz norm alone.