# Bayesian Inference with Shaped Deep Non-linear MLPs

arXiv: https://arxiv.org/abs/2605.30860 · HTML: https://mlsanity.com/paper/2605.30860/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 30 of 33)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [The Principles of Deep Learning Theory](https://mlsanity.com/paper/2106.10165/) (2021)
- [Finite Depth and Width Corrections to the Neural Tangent Kernel](https://mlsanity.com/paper/1909.05989/) (2019)
- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)
- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [Tensor Programs III: Neural Matrix Laws](https://mlsanity.com/paper/2009.10685/) (2020)
- [The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit](https://mlsanity.com/paper/2306.17759/) (2023)
- [The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization](https://mlsanity.com/paper/2206.02768/) (2022)
- [Predicting the outputs of finite deep neural networks trained with noisy gradients](https://mlsanity.com/paper/2004.01190/) (2020)
- [Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers](https://mlsanity.com/paper/2203.08120/) (2022)
- [Nonlinear spiked covariance matrices and signal propagation in deep neural networks](https://mlsanity.com/paper/2402.10127/) (2024)
- [Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies](https://mlsanity.com/paper/2204.01058/) (2022)
- [Bayesian Interpolation with Deep Linear Networks](https://mlsanity.com/paper/2212.14457/) (2022)
- [Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks](https://mlsanity.com/paper/2201.04753/) (2022)
- [Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes](https://mlsanity.com/paper/2312.11737/) (2023)
- [Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers](https://mlsanity.com/paper/2406.03260/) (2024)
- [Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime](https://mlsanity.com/paper/2505.03577/) (2025)
- [Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networks](https://mlsanity.com/paper/2306.05850/) (2023)

## Built on by (most cited first, 1 of 1)

- [Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product](https://mlsanity.com/paper/2606.30831/) (2026)

## Abstract (arXiv)

A central aim of deep learning theory is to characterize how neural networks make predictions in the regime of simultaneously large model and training set size. Since the limits of diverging number of model parameters and dataset size do not commute it is not clear a priori what limits exist. In this work, we shed new light on these questions by studying Bayesian inference in deep non-linear MLPs in the regime where the number of training samples (P), the input dimension (N0), the hidden layer width (N), and the number of hidden layers (L) can all be large. We build on the Neural Covariance SDE (Li et al., 2022) to analyze predictive posteriors in the regime where LP/N∈Theta(1), playing the role of an effective network depth. Our framework covers both smooth and ReLU activation functions and applies to arbitrary temperature. We find to first order in LP/N a simple criterion for which data generating processes benefit from depth in the sense that larger LP/N increases the Bayesian model evidence. We also give a novel derivation of a prior result from the physics literature that at least to first order in LP/N, the Bayesian predictive posterior is remarkably simple and is simply equivalent to that of a data-dependent kernel method.