Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime
Not on the map. Why?
Abstract (arXiv)
The scaling limit where both the size of the training set P and the width N of a deep neural network grow at the same rate, the so-called proportional-width regime, has been intensely studied for shallow, single-hidden-layer networks. However, extending these non-perturbative results from shallow architectures to deep non-linear networks has proven very challenging. Here we present an effective approximate approach to predict the generalization performance of Bayesian multi-layer perceptrons (MLPs) of fixed depth L on arbitrary high-dimensional data. We propose an equivalent Wishart Ansatz to capture the dominant stochastic fluctuations of the hierarchical empirical kernels of MLPs. This allows us to perform a large deviation analysis for the partition function of MLPs in the proportional limit, expressed in terms of a renormalized NNGP kernel. In this description, even strong representation learning in the proportional limit is encoded in at most L scalar order parameters, determined self-consistently. Extending the approach to convolutional architectures (CNNs), we identify a hierarchical local kernel renormalization mechanism, which allows to quantify more complex data-dependent transformations of the large-width kernel in CNNs due to finite-width effects. We test our effective theory against sampling experiments from the Bayesian posterior of finite deep neural networks with depths L ~ O(10) and P~ O(103) on classic benchmark datasets, finding overall very good agreement together with two distinct types of systematic deviations.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 51 of its 87 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
51 of 87 references.
References · 51
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
- 2019Fantastic Generalization Measures and Where to Find Themcited by 513
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
38 more
- 2016Deep Learning without Poor Local Minimacited by 234
- 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 191
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
- 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
- 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
- 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
- 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
- 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
- 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
- 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
- 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
- 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
- 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
- 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
- 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
- 2022Bayesian Interpolation with Deep Linear Networkscited by 21
- 2023Six Lectures on Linearized Neural Networkscited by 20
- 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
- 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
- 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 19
- 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 18
- 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 14
- 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
- 2022Globally Gated Deep Linear Networkscited by 12
- 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
- 2024Coding schemes in neural networks learning classification taskscited by 11
- 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
- 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
- 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
- 2023Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisationcited by 6
- 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
- 2025Predicting kernel regression learning curves from only raw data statisticscited by 6
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.