mlsanityData to 7 Oct 2026

Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation

arXiv 2501.18530 ↗First posted 30 Jan 2025stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width k and input dimension d are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size n scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters kd+k and of data n are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate n/d2, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 88 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

40 of 88 references.

References · 40

  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  5. 2021Deep learning: a statistical viewpointcited by 239
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  10. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  11. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  12. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
28 more
  1. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  2. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  3. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  4. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  5. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  6. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  7. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  8. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  9. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  10. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  11. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  12. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  13. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  14. 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
  15. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
  16. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  17. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 22
  18. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  19. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  20. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  21. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  22. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  23. 2024Coding schemes in neural networks learning classification taskscited by 11
  24. 2021Statistical limits of dictionary learning: random matrix theory and the spectral replica methodcited by 10
  25. 2013Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learningcited by 6
  26. 2020Beyond the storage capacity: data driven satisfiability transitioncited by 6
  27. 2024Random features and polynomial rulescited by 6
  28. 2024On the phase diagram of extensive-rank symmetric matrix denoising beyond rotational invariancecited by 4

Built on by

Not cited yet.