mlsanityData to 7 Oct 2026

Bayesian neural network unit priors and generalized Weibull-tail property

arXiv 2110.02885 ↗First posted 6 Oct 2021stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years. Hidden units are proven to follow a Gaussian process limit when the layer width tends to infinity. Recent work has suggested that finite Bayesian neural networks may outperform their infinite counterparts because they adapt their internal representations flexibly. To establish solid ground for future research on finite-width neural networks, our goal is to study the prior induced on hidden units. Our main result is an accurate description of hidden units tails which shows that unit priors become heavier-tailed going deeper, thanks to the introduced notion of generalized Weibull-tail. This finding sheds light on the behavior of hidden units of finite Bayesian neural networks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

16 of 30 references.

References · 16

  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  3. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  4. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  6. 2016Deep Information Propagationcited by 194
  7. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  8. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  9. 2021Bayesian Neural Network Priors Revisitedcited by 121
  10. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  11. 2020A statistical theory of cold posteriors in deep neural networkscited by 53
  12. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
4 more
  1. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  2. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  3. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  4. 2020Stable behaviour of infinitely wide deep neural networkscited by 23

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 1.

5 papers.

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  3. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  4. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  5. 2021Dependence between Bayesian neural network unitscited by 3