# Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information

arXiv: https://arxiv.org/abs/2505.17907 · HTML: https://mlsanity.com/paper/2505.17907/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 10 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 5 of 5)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks](https://mlsanity.com/paper/2111.15256/) (2021)

## Built on by (most cited first, 1 of 1)

- [Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weights](https://mlsanity.com/paper/2507.18555/) (2025)

## Abstract (arXiv)

Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.