mlsanityData to 14 Oct 2026

Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime

arXiv 2505.03577 ↗First posted 6 May 2025math.ST cond-mat.dis-nn cs.ITcited by 6

Not on the map. Why?

Abstract (arXiv)

We rigorously analyse fully-trained neural networks of arbitrary depth in the Bayesian optimal setting in the so-called proportional scaling regime where the number of training samples and width of the input and all inner layers diverge proportionally. We prove an information-theoretic equivalence between the Bayesian deep neural network model trained from data generated by a teacher with matching architecture, and a simpler model of optimal inference in a generalized linear model. This equivalence enables us to compute the optimal generalization error for deep neural networks in this regime. We thus prove the "deep Gaussian equivalence principle" conjectured in Cui et al. (2023) (arXiv:2302.00375). Our result highlights that in order to escape this "trivialisation" of deep neural networks (in the sense of reduction to a linear model) happening in the strongly overparametrized proportional regime, models trained from much more data have to be considered.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 46 of its 79 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

46 of 79 references.

References · 46

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2017Deep Neural Networks as Gaussian Processescited by 622
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  8. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  9. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  10. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  11. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  12. 2016Generalization Properties of Learning with Random Featurescited by 164
34 more
  1. 2019Linearized two-layers neural networks in high dimensioncited by 155
  2. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138
  3. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  4. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  5. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  6. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  9. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  10. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  11. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  12. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  13. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  14. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  15. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  16. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  17. 2018Entropy and mutual information in models of deep neural networkscited by 51
  18. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  19. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  20. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  21. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  22. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
  23. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  24. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  25. 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
  26. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  27. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  28. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 13
  29. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  30. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  31. 2020Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Trainingcited by 11
  32. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  33. 2024On the phase diagram of extensive-rank symmetric matrix denoising beyond rotational invariancecited by 4
  34. 2022Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergencecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  2. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  3. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  4. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  5. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  6. 2026Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networkscited by 0