mlsanityData to 7 Oct 2026

Nonlinear spiked covariance matrices and signal propagation in deep neural networks

arXiv 2402.10127 ↗First posted 15 Feb 2024stat.ML cs.LG math.PRcited by 25

Not on the map. Why?

Abstract (arXiv)

Many recent works have studied the eigenvalue spectrum of the Conjugate Kernel (CK) defined by the nonlinear feature map of a feedforward neural network. However, existing results only establish weak convergence of the empirical eigenvalue distribution, and fall short of providing precise quantitative characterizations of the ''spike'' eigenvalues and eigenvectors that often capture the low-dimensional signal structure of the learning problem. In this work, we characterize these signal eigenvalues and eigenvectors for a nonlinear version of the spiked covariance model, including the CK as a special case. Using this general result, we give a quantitative description of how spiked eigenstructure in the input data propagates through the hidden layers of a neural network with random weights. As a second application, we study a simple regime of representation learning where the weight matrix develops a rank-one signal component over training and characterize the alignment of the target function with the spike eigenvector of the CK on test data.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 34 of its 88 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

34 of 88 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017Deep Neural Networks as Gaussian Processescited by 617
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  5. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  6. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  7. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  8. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  9. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  10. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  11. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  12. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
22 more
  1. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  2. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  3. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  4. 2022Universality of empirical risk minimizationcited by 73
  5. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  6. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  7. 2021What can linearized neural networks actually say about generalization?cited by 49
  8. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  9. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  10. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 40
  11. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  12. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  13. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  14. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  15. 2021Covariate Shift in High-Dimensional Random Feature Regressioncited by 24
  16. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  17. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  18. 2023Spectral alignment of stochastic gradient descent for high-dimensional classification taskscited by 13
  19. 2022Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networkscited by 12
  20. 2022Quantitative deterministic equivalent of sample covariance matrices with a general dependence structurecited by 9
  21. 2023Universality for the global spectrum of random inner-product kernel matrices in the polynomial regimecited by 8
  22. 2023Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  2. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  3. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  4. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  5. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  6. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 8
  7. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7
  8. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 6
  9. 2024On the phase diagram of extensive-rank symmetric matrix denoising beyond rotational invariancecited by 4
  10. 2025Low Rank Gradients and Where to Find Themcited by 4
  11. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  12. 2026How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretabilitycited by 3
13 more
  1. 2026Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimizationcited by 3
  2. 2024Signal-Plus-Noise Decomposition of Nonlinear Spiked Random Matrix Modelscited by 2
  3. 2024Generalization for Least Squares Regression With Simple Spiked Covariancescited by 2
  4. 2025Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scalingcited by 2
  5. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  6. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  7. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 1
  8. 2025Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrixcited by 0
  9. 2026Noise Stability of Transformer Modelscited by 0
  10. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  11. 2026How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysiscited by 0
  12. 2026Mitigating Spurious Correlations with Memorization-Guided Dataset De-Biasingcited by 0
  13. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0