Large Deviations of Gaussian Neural Networks with ReLU activation
Not on the map. Why?
Abstract (arXiv)
We prove a large deviation principle for deep neural networks with Gaussian weights and at most linearly growing activation functions, such as ReLU. This generalises earlier work, in which bounded and continuous activation functions were considered. In practice, linearly growing activation functions such as ReLU are most commonly used. We furthermore simplify previous expressions for the rate function and provide a power-series expansions for the ReLU case.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 11 of its 25 references that are in the map. First posted on arXiv 27 May 2024.
Builds on
11 of 25 references.
References · 11
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2016Gaussian Error Linear Units (GELUs)cited by 3,502
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
- 2021The Principles of Deep Learning Theorycited by 173
- 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
- 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 25
- 2021Squareplus: A Softplus-Like Algebraic Rectifiercited by 19
Built on by
Citing papers per year: 2026: 3.
3 papers.