Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information
Not on the map. Why?
Abstract (arXiv)
Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 5 of its 10 references that are in the map. First posted on arXiv 23 May 2025.
Builds on
5 of 10 references.
References · 5
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
- 2021Deep learning: a statistical viewpointcited by 240
- 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
- 2021Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networkscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.