mlsanityData to 7 Oct 2026

Statistical mechanics of extensive-width Bayesian neural networks near interpolation

arXiv 2505.24849 ↗First posted 30 May 2025stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 9

Not on the map. Why?

Abstract (arXiv)

For three decades statistical mechanics has been providing a framework to analyse neural networks. However, the theoretically tractable models, e.g., perceptrons, random features models and kernel machines, or multi-index models and committee machines with few neurons, remained simple compared to those used in applications. In this paper we help reducing the gap between practical networks and their theoretical understanding through a statistical physics analysis of the supervised learning of a two-layer fully connected network with generic weight distribution and activation function, whose hidden layer is large but remains proportional to the inputs dimension. This makes it more realistic than infinitely wide networks where no feature learning occurs, but also more expressive than narrow ones or with fixed inner weights. We focus on the Bayes-optimal learning in the teacher-student scenario, i.e., with a dataset generated by another network with the same architecture. We operate around interpolation, where the number of trainable parameters and of data are comparable and feature learning emerges. Our analysis uncovers a rich phenomenology with various learning transitions as the number of data increases. In particular, the more strongly the features (i.e., hidden neurons of the target) contribute to the observed responses, the less data is needed to learn them. Moreover, when the data is scarce, the model only learns non-linear combinations of the teacher weights, rather than "specialising" by aligning its weights with the teacher's. Specialisation occurs only when enough data becomes available, but it can be hard to find for practical training algorithms, possibly due to statistical-to-computational~gaps.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 41 of its 88 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

41 of 88 references.

References · 41

  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  5. 2021Deep learning: a statistical viewpointcited by 239
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  10. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  11. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  12. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
29 more
  1. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  2. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  3. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  4. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  5. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  6. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  7. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  8. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  9. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  10. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  11. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  12. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  13. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  14. 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
  15. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
  16. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  17. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 22
  18. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  19. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  20. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  21. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  22. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  23. 2024Coding schemes in neural networks learning classification taskscited by 11
  24. 2021Statistical limits of dictionary learning: random matrix theory and the spectral replica methodcited by 10
  25. 2013Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learningcited by 6
  26. 2020Beyond the storage capacity: data driven satisfiability transitioncited by 6
  27. 2024Random features and polynomial rulescited by 6
  28. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  29. 2024On the phase diagram of extensive-rank symmetric matrix denoising beyond rotational invariancecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  2. 2025Generalization performance of narrow one-hidden layer networks in the teacher-student settingcited by 2
  3. 2026Statistical Properties of Training & Generalizationcited by 2
  4. 2026Factual recall in linear associative memories: sharp asymptotics and mechanistic insightscited by 1
  5. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  6. 2025Evaluating Double Descent in Machine Learning: Insights from Tree-Based Models Applied to a Genomic Prediction Taskcited by 0
  7. 2026Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networkscited by 0
  8. 2026Asymmetric Scaling Laws from Sparse Featurescited by 0
  9. 2026Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networkscited by 0