mlsanityData to 14 Oct 2026

Precise characterization of the prior predictive distribution of deep ReLU networks

arXiv 2106.06615 ↗First posted 11 Jun 2021cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture. Similar in spirit to the kind of analysis that has been developed to devise better initialization schemes for neural networks (cf. He- or Xavier initialization), we derive a precise characterization of the prior predictive distribution of finite-width ReLU networks with Gaussian weights. While theoretical results have been obtained for their heavy-tailedness, the full characterization of the prior predictive distribution (i.e. its density, CDF and moments), remained unknown prior to this work. Our analysis, based on the Meijer-G function, allows us to quantify the influence of architectural choices such as the width or depth of the network on the resulting shape of the prior predictive distribution. We also formally connect our results to previous work in the infinite width setting, demonstrating that the moments of the distribution converge to those of a normal log-normal mixture in the infinite depth limit. Finally, our results provide valuable guidance on prior design: for instance, controlling the predictive variance with depth- and width-informed priors on the weights of the network.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 20 of its 52 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

20 of 52 references.

References · 20

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  6. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  9. 2016Deep Information Propagationcited by 195
  10. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  11. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  12. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
8 more
  1. 2021Bayesian Neural Network Priors Revisitedcited by 121
  2. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  3. 2020The Case for Bayesian Deep Learningcited by 69
  4. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  5. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  6. 2020Deep kernel processescited by 34
  7. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  8. 2020Learning outside the Black-Box: The pursuit of interpretable modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 10; 2024: 4; 2025: 4; 2026: 1.

28 papers.

Sort:

  1. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  2. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  3. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  4. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  5. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  6. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  7. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  8. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  9. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  10. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  11. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  12. 2023Quantitative CLTs in Deep Neural Networkscited by 20
16 more
  1. 2023Structures of Neural Network Effective Theoriescited by 13
  2. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  3. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  4. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  5. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  6. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  7. 2021Bayesian neural network unit priors and generalized Weibull-tail propertycited by 5
  8. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  9. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
  10. 2021Dependence between Bayesian neural network unitscited by 3
  11. 2022An Empirical Analysis of the Advantages of Finite- v.s. Infinite-Width Bayesian Neural Networkscited by 1
  12. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  13. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  14. 2025CARE: Confidence-aware Ratio Estimation for Medical Biomarkerscited by 0
  15. 2025Simulation Priors for Data-Efficient Deep Learningcited by 0
  16. 2026Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performancecited by 0