The effect of Leaky ReLUs on the training and generalization of overparameterized networks
Not on the map. Why?
Abstract (arXiv)
We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, α. We show that α =-1, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 38 references that are in the map. First posted on arXiv 19 Feb 2024.
Builds on
16 of 38 references.
References · 16
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
- 2014Qualitatively characterizing neural network optimization problemscited by 186
- 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
- 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
4 more
- 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
- 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
- 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
- 2023Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimizationcited by 6
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.