High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercube
Not on the map. Why?
Abstract (arXiv)
We establish high-probability bounds for mixed input derivatives of wide random neural networks whose activation derivatives satisfy a factorial growth bound. Our main result specializes these estimates to tanh networks with Xavier initialization. A direct deterministic analysis based on Euclidean operator norms of the weight matrices yields derivative bounds that generally grow exponentially with the depth. We show that this growth can be substantially improved for sufficiently wide Gaussian networks by isolating the term that is linear in the highest-order derivative and controlling the corresponding tangent directions by measurable finite nets. For scalar-output tanh networks with Gaussian weights and Xavier initialization, we prove that there exist constants C,C0,C1>0 such that, whenever the common hidden width satisfies n ≥ Cleft(L3n02(1+log n0)+L2left(1+log(L/η)right)right), then, with probability at least 1-η, the estimate left|DuR_Phi(L)(x)right| ≤ C0 |u|! (C1L)|u|-1prodj∈ uβj(η,n0) holds simultaneously for every non-empty usubseteq[n0] and every x∈[0,1]n0. Thus, the first-order derivative bound is independent of the depth, while a square-free mixed derivative of order |u| grows at most polynomially as L|u|-1, apart from the coordinate factors. As consequences, we obtain high-probability bounds for the Euclidean Lipschitz constant and for weighted Sobolev norms of the network realization. The latter connect the derivative estimates to quasi-Monte Carlo integration and indicate how such regularity can enter the analysis of QMC-based training.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 27 Aug 2026.
Builds on
24 of 33 references.
References · 24
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2013Intriguing properties of neural networkscited by 5,155
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 293
- 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
- 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
- 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
- 2021On the approximation of functions by tanh neural networkscited by 81
- 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
- 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
12 more
- 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 25
- 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
- 2020Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequencescited by 20
- 2023Quantitative CLTs in Deep Neural Networkscited by 20
- 2021Large-width functional asymptotics for deep Gaussian neural networkscited by 15
- 2020Higher-order Quasi-Monte Carlo Training of Deep Neural Networkscited by 12
- 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 11
- 2020On Random Matrices Arising in Deep Neural Networks: General I.I.D. Casecited by 10
- 2023An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Networkcited by 3
- 2024Provable Bounds on the Hessian of Neural Networks: Derivative-Preserving Reachability Analysiscited by 3
- 2023Upper and lower bounds for the Lipschitz constant of random neural networkscited by 2
- 2026Lattice-based Deep Neural Networks: Regularity and Tailored Regularizationcited by 2
Built on by
Not cited yet.