mlsanityData to 14 Oct 2026

Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation

arXiv 2510.24616 ↗First posted 28 Oct 2025stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 0

Not on the map. Why?

Abstract (arXiv)

For four decades statistical physics has been providing a framework to analyse neural networks. A long-standing question remained on its capacity to tackle deep learning models capturing rich feature learning effects, thus going beyond the narrow networks or kernel methods analysed until now. We positively answer through the study of the supervised learning of a multi-layer perceptron. Importantly, (i) its width scales as the input dimension, making it more prone to feature learning than ultra wide networks, and more expressive than narrow ones or ones with fixed embedding layers; and (ii) we focus on the challenging interpolation regime where the number of trainable parameters and data are comparable, which forces the model to adapt to the task. We consider the matched teacher-student setting. Therefore, we provide the fundamental limits of learning random deep neural network targets and identify the sufficient statistics describing what is learnt by an optimally trained network as the data budget increases. A rich phenomenology emerges with various learning transitions. With enough data, optimal performance is attained through the model's "specialisation" towards the target, but it can be hard to reach for training algorithms which get attracted by sub-optimal solutions predicted by the theory. Specialisation occurs inhomogeneously across layers, propagating from shallow towards deep ones, but also across neurons in each layer. Furthermore, deeper targets are harder to learn. Despite its simplicity, the Bayes-optimal setting provides insights on how the depth, non-linearity and finite (proportional) width influence neural networks in the feature learning regime that are potentially relevant in much more general settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 88 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

41 of 88 references.

References · 41

  1. 2017Deep Neural Networks as Gaussian Processescited by 622
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  5. 2021Deep learning: a statistical viewpointcited by 240
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  10. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  11. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  12. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
29 more
  1. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  2. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  3. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  4. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  5. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  6. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
  7. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  8. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  9. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  10. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  11. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  12. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  13. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 26
  14. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  15. 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
  16. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
  17. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  18. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  19. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  20. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 19
  21. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 13
  22. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  23. 2024Coding schemes in neural networks learning classification taskscited by 11
  24. 2021Statistical limits of dictionary learning: random matrix theory and the spectral replica methodcited by 10
  25. 2013Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learningcited by 6
  26. 2020Beyond the storage capacity: data driven satisfiability transitioncited by 6
  27. 2024Random features and polynomial rulescited by 6
  28. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  29. 2024On the phase diagram of extensive-rank symmetric matrix denoising beyond rotational invariancecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolation
  2. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models
  3. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
  4. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
  5. 2024Collective variables of neural networks: empirical time evolution and scaling laws
  6. 2023Spatially heterogeneous learning by a deep student machine
  7. 2026There Will Be a Scientific Theory of Deep Learning
  8. 2025Theoretical Compression Bounds for Wide Multilayer Perceptrons