# On neural network kernels and the storage capacity problem

arXiv: https://arxiv.org/abs/2201.04669 · HTML: https://mlsanity.com/paper/2201.04669/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 12 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity](https://mlsanity.com/paper/1602.05897/) (2016)
- [Exact posterior distributions of wide Bayesian neural networks](https://mlsanity.com/paper/2006.10541/) (2020)
- [Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations](https://mlsanity.com/paper/1907.07578/) (2019)
- [Effect of Activation Functions on the Training of Overparametrized Neural Nets](https://mlsanity.com/paper/1908.05660/) (2019)
- [Activation function dependence of the storage capacity of treelike neural networks](https://mlsanity.com/paper/2007.11136/) (2020)

## Built on by (most cited first, 2 of 2)

- [Long Sequence Hopfield Memory](https://mlsanity.com/paper/2306.04532/) (2023)
- [Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks](https://mlsanity.com/paper/2410.06717/) (2024)

## Abstract (arXiv)

In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.