mlsanityData to 7 Oct 2026

Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks

arXiv 2201.04753 ↗First posted 13 Jan 2022math.PR cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper is concerned with the asymptotic distribution of the largest eigenvalues for some nonlinear random matrix ensemble stemming from the study of neural networks. More precisely we consider M= frac1m YY^top with Y=f(WX) where W and X are random rectangular matrices with i.i.d. centered entries. This models the data covariance matrix or the Conjugate Kernel of a single layered random Feed-Forward Neural Network. The function f is applied entrywise and can be seen as the activation function of the neural network. We show that the largest eigenvalue has the same limit (in probability) as that of some well-known linear random matrix ensembles. In particular, we relate the asymptotic limit of the largest eigenvalue for the nonlinear model to that of an information-plus-noise random matrix, establishing a possible phase transition depending on the function f and the distribution of W and X. This may be of interest for applications to machine learning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 13 of its 28 references that are in the map. First posted on arXiv 13 Jan 2022.

Builds on

13 of 28 references.

References · 13

  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2014The Loss Surfaces of Multilayer Networkscited by 278
  5. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  6. 2017A Random Matrix Approach to Neural Networkscited by 80
  7. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  8. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  9. 2020Neural Kernels Without Tangentscited by 56
  10. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  11. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  12. 2020Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysiscited by 14
1 more
  1. 2021Random matrices in service of ML footprint: ternary random features with no performance losscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 2; 2025: 2; 2026: 2.

12 papers.

  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  3. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  4. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  5. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  6. 2023Universality for the global spectrum of random inner-product kernel matrices in the polynomial regimecited by 8
  7. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  8. 2023Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networkscited by 5
  9. 2025Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scalingcited by 2
  10. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  11. 2025Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrixcited by 0
  12. 2026Spectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantizationcited by 0