mlsanityData to 7 Oct 2026

More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize

arXiv 2203.06176 ↗First posted 11 Mar 2022cs.LG stat.MLcited by 87

Not on the map. Why?

Abstract (arXiv)

Of theories for why large-scale machine learning models generalize despite being vastly overparameterized, which of their assumptions are needed to capture the qualitative phenomena of generalization in the real world? On one hand, we find that most theoretical analyses fall short of capturing these qualitative phenomena even for kernel regression, when applied to kernels derived from large-scale neural networks (e.g., ResNet-50) and real data (e.g., CIFAR-100). On the other hand, we find that the classical GCV estimator (Craven and Wahba, 1978) accurately predicts generalization risk even in such overparameterized settings. To bolster this empirical finding, we prove that the GCV estimator converges to the generalization risk whenever a local random matrix law holds. Finally, we apply this random matrix theory lens to explain why pretrained representations generalize better as well as what factors govern scaling laws for kernel regression. Our findings suggest that random matrix theory, rather than just being a toy model, may be central to understanding the properties of neural representations in practice.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 36 of its 69 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

36 of 69 references.

References · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2019Benign Overfitting in Linear Regressioncited by 544
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
24 more
  1. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  2. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  3. 2021Explaining Neural Scaling Lawscited by 353
  4. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  5. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  6. 2019Two models of double descent for weak featurescited by 194
  7. 2018To understand deep learning we need to understand kernel learningcited by 186
  8. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  10. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  11. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  12. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  13. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  14. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  15. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  16. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  17. 2017Implicit Regularization in Deep Learningcited by 90
  18. 2020In Search of Robust Measures of Generalizationcited by 79
  19. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  20. 2021Cross-validation: what does it estimate and how well does it do it?cited by 65
  21. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  22. 2020Implicit Regularization of Random Feature Modelscited by 54
  23. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  24. 2017From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 17; 2024: 34; 2025: 19; 2026: 6.

87 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  3. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  4. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  5. 2022A Solvable Model of Neural Scaling Lawscited by 101
  6. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  7. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  8. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  9. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  10. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  11. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
  12. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
38 more
  1. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  2. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  3. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  4. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  5. 2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
  6. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  7. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  8. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  9. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  10. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  11. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
  12. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 21
  13. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  14. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 18
  15. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  16. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  17. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  18. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  19. 2024Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictionscited by 13
  20. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  21. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 13
  22. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  23. 2023Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validationcited by 11
  24. 2023Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuningcited by 11
  25. 2023Sample based Explanations via Generalized Representerscited by 10
  26. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  27. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  28. 2023The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasetscited by 9
  29. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  30. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  31. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  32. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7
  33. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  34. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  35. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  36. 2022A picture of the space of typical learnable taskscited by 6
  37. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  38. 2023MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Modelscited by 6

50 most cited of 87 All 87 (JSON)