Generalization for Least Squares Regression With Simple Spiked Covariances
Not on the map. Why?
Abstract (arXiv)
Random matrix theory has proven to be a valuable tool in analyzing the generalization of linear models. However, the generalization properties of even two-layer neural networks trained by gradient descent remain poorly understood. To understand the generalization performance of such networks, it is crucial to characterize the spectrum of the feature matrix at the hidden layer. Recent work has made progress in this direction by describing the spectrum after a single gradient step, revealing a spiked covariance structure. Yet, the generalization error for linear models with spiked covariances has not been previously determined. This paper addresses this gap by examining two simple models exhibiting spiked covariances. We derive their generalization error in the asymptotic proportional regime. Our analysis demonstrates that the eigenvector and eigenvalue corresponding to the spike significantly influence the generalization error.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 17 Oct 2024.
Builds on
27 of 45 references.
References · 27
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
- 2017High-dimensional dynamics of generalization error in neural networkscited by 241
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
- 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
- 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
- 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
- 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
- 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
- 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
15 more
- 2020Implicit Regularization of Random Feature Modelscited by 54
- 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
- 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
- 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
- 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
- 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 26
- 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 26
- 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
- 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
- 2013OptShrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkagecited by 14
- 2023Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoiserscited by 14
- 2021Analysis of One-Hidden-Layer Neural Networks via the Resolvent Methodcited by 9
- 2023Learning Hierarchical Polynomials with Three-Layer Neural Networkscited by 9
- 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 8
- 2023Least Squares Regression Can Exhibit Under-Parameterized Double Descentcited by 4
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.