Upper and lower bounds for the Lipschitz constant of random neural networks
Not on the map. Why?
Abstract (arXiv)
Empirical studies have widely demonstrated that neural networks are highly sensitive to small, adversarial perturbations of the input. The worst-case robustness against these so-called adversarial examples can be quantified by the Lipschitz constant of the neural network. In this paper, we study upper and lower bounds for the Lipschitz constant of random ReLU neural networks. Specifically, we assume that the weights and biases follow a generalization of the He initialization, where general symmetric distributions for the biases are permitted. For deep networks of fixed depth and sufficiently large width, our established upper bound is larger than the lower bound by a factor that is logarithmic in the width. In contrast, for shallow neural networks we characterize the Lipschitz constant up to an absolute numerical constant that is independent of all parameters.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 2 Nov 2023.
Builds on
34 of 44 references.
References · 34
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2013Intriguing properties of neural networkscited by 5,114
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Adversarial examples in the physical worldcited by 1,762
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
- 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
- 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
- 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
22 more
- 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
- 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
- 2021A Universal Law of Robustness via Isoperimetrycited by 150
- 2018Towards the first adversarially robust neural network model on MNISTcited by 104
- 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
- 2020Adversarial Distributional Training for Robust Deep Learningcited by 79
- 2020A law of robustness for two-layers neural networkscited by 54
- 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
- 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
- 2022A Unified Wasserstein Distributional Robustness Framework for Adversarial Trainingcited by 36
- 2020Deep Networks and the Multiple Manifold Problemcited by 35
- 2019A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distancecited by 32
- 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
- 2021A single gradient step finds adversarial examples on random two-layers neural networkscited by 26
- 2021Adversarial Examples in Multi-Layer Random ReLU Networkscited by 26
- 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
- 2021The Separation Capacity of Random Neural Networkscited by 14
- 2022Adversarial Examples in Random Neural Networks with General Activationscited by 13
- 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
- 2019Towards a regularity theory for ReLU networks -- chain rule and global error estimatescited by 8
- 2022Adversarial Robustness is at Odds with Lazy Trainingcited by 6
- 2023Local Lipschitz Constant Computation of ReLU-FNNs: Upper Bound Computation with Exactness Verificationcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.