mlsanityData to 7 Oct 2026

Learning curves for deep structured Gaussian feature models

arXiv 2303.00564 ↗First posted 1 Mar 2023stat.ML cond-mat.dis-nn cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In recent years, significant attention in deep learning theory has been devoted to analyzing when models that interpolate their training data can still generalize well to unseen examples. Many insights have been gained from studying models with multiple layers of Gaussian random features, for which one can compute precise generalization asymptotics. However, few works have considered the effect of weight anisotropy; most assume that the random features are generated using independent and identically distributed Gaussian weights, and allow only for structure in the input data. Here, we use the replica trick from statistical physics to derive learning curves for models with many layers of structured Gaussian features. We show that allowing correlations between the rows of the first layer of features can aid generalization, while structure in later layers is generally detrimental. Our results shed light on how weight structure affects generalization in a simple class of solvable models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

33 of 57 references.

References · 33

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2021Explaining Neural Scaling Lawscited by 353
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
21 more
  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  2. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  3. 2022A Solvable Model of Neural Scaling Lawscited by 101
  4. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  5. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  6. 2022Universality of empirical risk minimizationcited by 73
  7. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  8. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  9. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  10. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  11. 2020Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized modelscited by 26
  12. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  13. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  14. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  15. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  16. 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
  17. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  18. 2021Depth induces scale-averaging in overparameterized linear Bayesian neural networkscited by 7
  19. 2022Bias-variance decomposition of overparameterized regression with random linear featurescited by 7
  20. 2020Strong replica symmetry for high-dimensional disordered log-concave Gibbs measurescited by 6
  21. 2023How does training shape the Riemannian geometry of neural network representations?cited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 9; 2025: 3.

13 papers.

Sort:

By area:

  1. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  2. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  3. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  4. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  5. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  6. 2024High-dimensional learning of narrow neural networkscited by 12
  7. 2024Risk and cross validation in ridge regression with correlated samplescited by 8
  8. 2024Random features and polynomial rulescited by 6
  9. 2025Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Modelscited by 6
  10. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  11. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  12. 2024A solvable generative model with a linear, one-step denoisercited by 2
1 more
  1. 2025One-Bit Quantization for Random Features Modelscited by 0