mlsanityData to 14 Oct 2026

The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks

arXiv 2505.17958 ↗First posted 23 May 2025stat.ML cond-mat.dis-nn cs.ITcited by 11

Not on the map. Why?

Abstract (arXiv)

We study the high-dimensional asymptotics of empirical risk minimization (ERM) in over-parametrized two-layer neural networks with quadratic activations trained on synthetic data. We derive sharp asymptotics for both training and test errors by mapping the ℓ2-regularized learning problem to a convex matrix sensing task with nuclear norm penalization. This reveals that capacity control in such networks emerges from a low-rank structure in the learned feature maps. Our results characterize the global minima of the loss and yield precise generalization thresholds, showing how the width of the target function governs learnability. This analysis bridges and extends ideas from spin-glass methods, matrix factorization, and convex optimization and emphasizes the deep link between low-rank matrix sensing and learning in quadratic neural networks.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 51 of its 99 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

51 of 99 references.

References · 51

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  3. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  4. 2017Deep Neural Networks as Gaussian Processescited by 622
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2017Implicit Regularization in Matrix Factorizationcited by 322
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2019Two models of double descent for weak featurescited by 195
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2017A Rewriting System for Convex Optimization Problemscited by 179
  12. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
38 more
  1. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  2. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  3. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  4. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  5. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  6. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  7. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  8. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  9. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  10. 2022Universality of empirical risk minimizationcited by 75
  11. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  12. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  13. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  14. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
  15. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  16. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  17. 2021A unifying tutorial on Approximate Message Passingcited by 50
  18. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 44
  19. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  20. 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
  21. 2020Phase retrieval in high dimensions: Statistical and computational phase transitionscited by 39
  22. 2021Graph-based Approximate Message Passing Iterationscited by 35
  23. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  24. 2017Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimationcited by 26
  25. 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 25
  26. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  27. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  28. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 16
  29. 2025Survey on Algorithms for multi-index modelscited by 15
  30. 2025Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formulacited by 14
  31. 2019Stationary Points of Shallow Neural Networks with Quadratic Activation Functioncited by 12
  32. 2025Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholdscited by 12
  33. 2025Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applicationscited by 11
  34. 2023Universality for the global spectrum of random inner-product kernel matrices in the polynomial regimecited by 10
  35. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  36. 2023Fitting an ellipsoid to random points: predictions using the replica methodcited by 7
  37. 2023Exact threshold for approximate ellipsoid fitting of random pointscited by 7
  38. 2023Universality of max-margin classifierscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  3. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 9
  4. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 6
  5. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  6. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  7. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  8. 2026Factual recall in linear associative memories: sharp asymptotics and mechanistic insightscited by 1
  9. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  10. 2026The sharp SAT/UNSAT phase transition in random ellipsoid fittingcited by 1
  11. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0