mlsanityData to 7 Oct 2026

Bayes-optimal Learning of Deep Random Networks of Extensive-width

arXiv 2302.00375 ↗First posted 1 Feb 2023stat.ML cond-mat.dis-nn cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width are proportionally large. We propose a closed-form expression for the Bayes-optimal test error, for regression and classification tasks. We further compute closed-form expressions for the test errors of ridge regression, kernel and random features regression. We find, in particular, that optimally regularized ridge regression, as well as kernel regression, achieve Bayes-optimal performances, while the logistic loss yields a near-optimal test error for classification. We further show numerically that when the number of samples grows faster than the dimension, ridge and kernel methods become suboptimal, while neural networks achieve test error close to zero from quadratically many samples.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

41 of 72 references.

References · 41

  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  2. 2017Deep Neural Networks as Gaussian Processescited by 617
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2021The Principles of Deep Learning Theorycited by 169
  7. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  8. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  9. 2019Linearized two-layers neural networks in high dimensioncited by 155
  10. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  11. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
29 more
  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  2. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  3. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  4. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  5. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  6. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  7. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  8. 2017A Random Matrix Approach to Neural Networkscited by 80
  9. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  10. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  11. 2022Universality of empirical risk minimizationcited by 73
  12. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  13. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  14. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  15. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  16. 2020Phase retrieval in high dimensions: Statistical and computational phase transitionscited by 39
  17. 2020Exact posterior distributions of wide Bayesian neural networkscited by 29
  18. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  19. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  20. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  21. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 18
  22. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  23. 2019Mean-field inference methods for neural networkscited by 16
  24. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 15
  25. 2016Statistical Mechanics of High-Dimensional Inferencecited by 13
  26. 2021On the interplay between data structure and loss function in classification problemscited by 12
  27. 2022Decomposing neural networks as mappings of correlation functionscited by 12
  28. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  29. 2019Large deviations for the perceptron model and consequences for active learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 13; 2025: 14; 2026: 5.

41 papers.

Sort:

By area:

  1. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  2. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  3. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  4. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  5. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  6. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  7. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
  8. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  9. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  10. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  11. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  12. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
29 more
  1. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  2. 2024High-dimensional learning of narrow neural networkscited by 12
  3. 2024Coding schemes in neural networks learning classification taskscited by 11
  4. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  5. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  6. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  7. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  8. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7
  9. 2023Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamicscited by 6
  10. 2024Random features and polynomial rulescited by 6
  11. 2024Restoring balance: principled under/oversampling of data for optimal classificationcited by 6
  12. 2024Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networkscited by 6
  13. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  14. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  15. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
  16. 2024Asymptotic generalization error of a single-layer graph convolutional networkcited by 3
  17. 2025Bayes optimal learning of attention-indexed modelscited by 3
  18. 2025A statistical physics framework for optimal learningcited by 3
  19. 2025Statistical physics analysis of graph neural networks: Approaching optimality in the contextual stochastic block modelcited by 2
  20. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 2
  21. 2026A unified theory of feature learning in RNNs and DNNscited by 1
  22. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  23. 2023Spatially heterogeneous learning by a deep student machinecited by 0
  24. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  25. 2025Evidence of Replica Symmetry Breaking under the Nishimori conditions in epidemic inference on graphscited by 0
  26. 2025One-Bit Quantization for Random Features Modelscited by 0
  27. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  28. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 0
  29. 2026Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networkscited by 0