mlsanityData to 7 Oct 2026

On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector

arXiv 2206.13646 ↗First posted 27 Jun 2022cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

It is an elementary fact in the scientific literature that the Lipschitz norm of the realization function of a feedforward fully-connected rectified linear unit (ReLU) artificial neural network (ANN) can, up to a multiplicative constant, be bounded from above by sums of powers of the norm of the ANN parameter vector. Roughly speaking, in this work we reveal in the case of shallow ANNs that the converse inequality is also true. More formally, we prove that the norm of the equivalence class of ANN parameter vectors with the same realization function is, up to a multiplicative constant, bounded from above by the sum of powers of the Lipschitz norm of the ANN realization function (with the exponents 1/2 and 1 ). Moreover, we prove that this upper bound only holds when employing the Lipschitz norm but does neither hold for H"older norms nor for Sobolev-Slobodeckij norms. Furthermore, we prove that this upper bound only holds for sums of powers of the Lipschitz norm with the exponents 1/2 and 1 but does not hold for the Lipschitz norm alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

14 of 21 references.

References · 14

  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2016An overview of gradient descent optimization algorithmscited by 1,021
  4. 2018Stochastic subgradient method converges on tame functionscited by 120
  5. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  6. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  7. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  8. 2019Full error analysis for the training of deep neural networkscited by 17
  9. 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
  10. 2019Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Non-Convex Optimizationcited by 12
  11. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  12. 2016Unifying abstract inexact convergence theorems and block coordinate variable metric iPianocited by 10
2 more
  1. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  2. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9

Built on by

Not cited yet.