mlsanityData to 7 Oct 2026

Fundamental limits of overparametrized shallow neural networks for supervised learning

arXiv 2307.05635 ↗First posted 11 Jul 2023cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 12

Not on the map. Why?

Abstract (arXiv)

We carry out an information-theoretical analysis of a two-layer neural network trained from input-output pairs generated by a teacher network with matching architecture, in overparametrized regimes. Our results come in the form of bounds relating i) the mutual information between training data and network weights, or ii) the Bayes-optimal generalization error, to the same quantities but for a simpler (generalized) linear model for which explicit expressions are rigorously known. Our bounds, which are expressed in terms of the number of training samples, input dimension and number of hidden units, thus yield fundamental performance limits for any neural network (and actually any learning procedure) trained from limited data generated according to our two-layer teacher neural network model. The proof relies on rigorous tools from spin glasses and is guided by ``Gaussian equivalence principles'' lying at the core of numerous recent analyses of neural networks. With respect to the existing literature, which is either non-rigorous or restricted to the case of the learning of the readout weights only, our results are information-theoretic (i.e. are not specific to any learning algorithm) and, importantly, cover a setting where all the network parameters are trained.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 58 of its 113 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

58 of 113 references.

References · 58

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  12. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
38 more
  1. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  2. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  6. 2016Generalization Properties of Learning with Random Featurescited by 163
  7. 2019Linearized two-layers neural networks in high dimensioncited by 155
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  10. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  11. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  12. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  13. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  14. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  15. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  16. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  17. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  18. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  19. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  20. 2017A Random Matrix Approach to Neural Networkscited by 80
  21. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  22. 2022Universality of empirical risk minimizationcited by 73
  23. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  24. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  25. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  26. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  27. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  28. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  29. 2018Entropy and mutual information in models of deep neural networkscited by 50
  30. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  31. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  32. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  33. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  34. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  35. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
  36. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  37. 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
  38. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 5; 2026: 1.

12 papers.

  1. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  2. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  3. 2024High-dimensional learning of narrow neural networkscited by 12
  4. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  5. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  6. 2023Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfittingcited by 3
  7. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
  8. 2024On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networkscited by 1
  9. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  10. 2025Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learningcited by 0
  11. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  12. 2026Teacher Geometry Shapes Learnability in Teacher-Student Networkscited by 0