mlsanityData to 7 Oct 2026

Bayesian Interpolation with Deep Linear Networks

arXiv 2212.14457 ↗First posted 29 Dec 2022stat.ML cs.LG math.PRcited by 21

Not on the map. Why?

Abstract (arXiv)

Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We give here a complete solution in the special case of linear networks with output dimension one trained using zero noise Bayesian inference with Gaussian weight priors and mean squared error as a negative log-likelihood. For any training dataset, network depth, and hidden layer widths, we find non-asymptotic expressions for the predictive posterior and Bayesian model evidence in terms of Meijer-G functions, a class of meromorphic special functions of a single complex variable. Through novel asymptotic expansions of these Meijer-G functions, a rich new picture of the joint role of depth, width, and dataset size emerges. We show that linear networks make provably optimal predictions at infinite depth: the posterior of infinitely deep linear networks with data-agnostic priors is the same as that of shallow networks with evidence-maximizing data-dependent priors. This yields a principled reason to prefer deeper networks when priors are forced to be data-agnostic. Moreover, we show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth, elucidating the salutary role of increased depth for model selection. Underpinning our results is a novel emergent notion of effective depth, given by the number of hidden layers times the number of data points divided by the network width; this determines the structure of the posterior in the large-data limit.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 61 of its 101 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

61 of 101 references.

References · 61

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2019Benign Overfitting in Linear Regressioncited by 544
38 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  2. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2021Explaining Neural Scaling Lawscited by 353
  7. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  8. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  10. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  11. 2016Deep Learning without Poor Local Minimacited by 233
  12. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  13. 2019Two models of double descent for weak featurescited by 194
  14. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  15. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  16. 2021The Principles of Deep Learning Theorycited by 169
  17. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  18. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  19. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  20. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  21. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  22. 2022A Solvable Model of Neural Scaling Lawscited by 101
  23. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  24. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  25. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  26. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  27. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  28. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  29. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  30. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  31. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  32. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  33. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  34. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  35. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  36. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  37. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  38. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 6; 2025: 3; 2026: 4.

21 papers.

Sort:

  1. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  2. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  3. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  4. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  5. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  6. 2023Structures of Neural Network Effective Theoriescited by 13
  7. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  8. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  9. 2024High-dimensional learning of narrow neural networkscited by 12
  10. 2024Coding schemes in neural networks learning classification taskscited by 11
  11. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  12. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
9 more
  1. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  2. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  3. 2024Proportional infinite-width infinite-depth limit for deep linear neural networkscited by 3
  4. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  5. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  6. 2026Revisiting the Volume Hypothesiscited by 1
  7. 2023Spectral-Bias and Kernel-Task Alignment in Physically Informed Neural Networkscited by 0
  8. 2025Deep Neural Nets as Hamiltonianscited by 0
  9. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0