MLPs at the EOC: Spectrum of the NTK

We study the properties of the Neural Tangent Kernel (NTK) oversetscriptscriptstyle∞K : ℝm0 × ℝm0 → ℝml × ml corresponding to infinitely wide l-layer Multilayer Perceptrons (MLPs) taking inputs from ℝm0 to outputs in ℝml equipped with activation functions φ(s) = a s + b vert s vert for some a,b ∈ ℝ and initialized at the Edge Of Chaos (EOC). We find that the entries oversetscriptscriptstyle∞K(x1,x2) can be approximated by the inverses of the cosine distances of the activations corresponding to x1 and x2 increasingly better as the depth l increases. By quantifying these inverse cosine distances and the spectrum of the matrix containing them, we obtain tight spectral bounds for the NTK matrix oversetscriptscriptstyle∞K = [frac1n oversetscriptscriptstyle∞K(xi1,xi2) : i1, i2 ∈ [1:n]] over a dataset x1,cdots,xn subset ℝm0, transferred from the inverse cosine distance matrix via our approximation result. Our results show that Deltaφ = fracb2a2+b2 determines the rate at which the condition number of the NTK matrix converges to its limit as depth increases, implying in particular that the absolute value (Deltaφ=1) is better than the ReLU (Deltaφ=frac12) in this regard.