# High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercube

arXiv: https://arxiv.org/abs/2608.26526 · HTML: https://mlsanity.com/paper/2608.26526/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [Exactly Computing the Local Lipschitz Constant of ReLU Networks](https://mlsanity.com/paper/2003.01219/) (2020)
- [Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?](https://mlsanity.com/paper/1801.03744/) (2018)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [Random Neural Networks in the Infinite Width Limit as Gaussian Processes](https://mlsanity.com/paper/2107.01562/) (2021)
- [Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks](https://mlsanity.com/paper/2203.07379/) (2022)
- [Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies](https://mlsanity.com/paper/2204.01058/) (2022)
- [Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences](https://mlsanity.com/paper/2005.12564/) (2020)
- [Quantitative CLTs in Deep Neural Networks](https://mlsanity.com/paper/2307.06092/) (2023)
- [Large-width functional asymptotics for deep Gaussian neural networks](https://mlsanity.com/paper/2102.10307/) (2021)
- [Higher-order Quasi-Monte Carlo Training of Deep Neural Networks](https://mlsanity.com/paper/2009.02713/) (2020)
- [Gaussian random field approximation via Stein's method with applications to wide random neural networks](https://mlsanity.com/paper/2306.16308/) (2023)
- [On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case](https://mlsanity.com/paper/2011.11439/) (2020)
- [An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network](https://mlsanity.com/paper/2312.03386/) (2023)
- [Provable Bounds on the Hessian of Neural Networks: Derivative-Preserving Reachability Analysis](https://mlsanity.com/paper/2406.04476/) (2024)
- [Upper and lower bounds for the Lipschitz constant of random neural networks](https://mlsanity.com/paper/2311.01356/) (2023)
- [Lattice-based Deep Neural Networks: Regularity and Tailored Regularization](https://mlsanity.com/paper/2603.02809/) (2026)

## Abstract (arXiv)

We establish high-probability bounds for mixed input derivatives of wide random neural networks whose activation derivatives satisfy a factorial growth bound. Our main result specializes these estimates to tanh networks with Xavier initialization. A direct deterministic analysis based on Euclidean operator norms of the weight matrices yields derivative bounds that generally grow exponentially with the depth. We show that this growth can be substantially improved for sufficiently wide Gaussian networks by isolating the term that is linear in the highest-order derivative and controlling the corresponding tangent directions by measurable finite nets. For scalar-output tanh networks with Gaussian weights and Xavier initialization, we prove that there exist constants C,C0,C1>0 such that, whenever the common hidden width satisfies n ≥ Cleft(L3n02(1+log n0)+L2left(1+log(L/η)right)right), then, with probability at least 1-η, the estimate left|DuR_Phi(L)(x)right| ≤ C0 |u|! (C1L)|u|-1prodj∈ uβj(η,n0) holds simultaneously for every non-empty usubseteq[n0] and every x∈[0,1]n0. Thus, the first-order derivative bound is independent of the depth, while a square-free mixed derivative of order |u| grows at most polynomially as L|u|-1, apart from the coordinate factors. As consequences, we obtain high-probability bounds for the Euclidean Lipschitz constant and for weighted Sobolev norms of the network realization. The latter connect the derivative estimates to quasi-Monte Carlo integration and indicate how such regularity can enter the analysis of QMC-based training.