Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks
Not on the map. Why?
Abstract (arXiv)
In recent years, the neural tangent kernel (NTK) and neural network Gaussian process kernel (NNGP) have given theoreticians tractable limiting cases of fully connected neural networks. However, the property of these kernels are poorly understood for activation functions other than powers of the ReLU. Our main contribution is a characterization of the RKHS of these kernels for activation functions whose only non-smoothness is at zero. This extends existing theory to numerous commonly used activation functions such as SELU, ELU, or LeakyReLU. Additionally, we analyze a broad set of special cases such as missing biases, two-layer networks, or polynomial activations. Our results show that a broad class of not infinitely smooth activations generate equivalent RKHSs at different network depths, depending only on the degree of the non-smoothness up to equivalence. On the other hand, the RKHS generated by polynomial activations depends on the network depth. Finally, we derive results for the smoothness of NNGP sample paths, characterizing the smoothness of infinitely wide neural networks at initialization.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 27 Jun 2025.
Builds on
40 of 56 references.
References · 40
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018On the Spectral Bias of Neural Networkscited by 1,341
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2018To understand deep learning we need to understand kernel learningcited by 187
- 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
- 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
28 more
- 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
- 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
- 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
- 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
- 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
- 2024Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Datacited by 92
- 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
- 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
- 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
- 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
- 2021What can linearized neural networks actually say about generalization?cited by 49
- 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 35
- 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 31
- 2023Generalization Ability of Wide Neural Networks on ℝcited by 30
- 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
- 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
- 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 21
- 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
- 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
- 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 16
- 2019Effect of Activation Functions on the Training of Overparametrized Neural Netscited by 14
- 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
- 2021Reverse Engineering the Neural Tangent Kernelcited by 12
- 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
- 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
- 2022Sobolev Spaces, Kernels and Discrepancies over Hyperspherescited by 6
- 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
- 2021Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysiscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.