# MLPs at the EOC: Spectrum of the NTK

arXiv: https://arxiv.org/abs/2501.13225 · HTML: https://mlsanity.com/paper/2501.13225/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2025.

## Built on by (most cited first, 1 of 1)

- [The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?](https://mlsanity.com/paper/2605.20749/) (2026)

## Abstract (arXiv)

We study the properties of the Neural Tangent Kernel (NTK) oversetscriptscriptstyle∞K : ℝm0 × ℝm0 → ℝml × ml corresponding to infinitely wide l-layer Multilayer Perceptrons (MLPs) taking inputs from ℝm0 to outputs in ℝml equipped with activation functions φ(s) = a s + b vert s vert for some a,b ∈ ℝ and initialized at the Edge Of Chaos (EOC). We find that the entries oversetscriptscriptstyle∞K(x1,x2) can be approximated by the inverses of the cosine distances of the activations corresponding to x1 and x2 increasingly better as the depth l increases. By quantifying these inverse cosine distances and the spectrum of the matrix containing them, we obtain tight spectral bounds for the NTK matrix oversetscriptscriptstyle∞K = [frac1n oversetscriptscriptstyle∞K(xi1,xi2) : i1, i2 ∈ [1:n]] over a dataset x1,cdots,xn subset ℝm0, transferred from the inverse cosine distance matrix via our approximation result. Our results show that Deltaφ = fracb2a2+b2 determines the rate at which the condition number of the NTK matrix converges to its limit as depth increases, implying in particular that the absolute value (Deltaφ=1) is better than the ReLU (Deltaφ=frac12) in this regard.