mlsanityData to 7 Oct 2026

Gibbs Sampling the Posterior of Neural Networks

arXiv 2306.02729 ↗First posted 5 Jun 2023cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we study sampling from a posterior derived from a neural network. We propose a new probabilistic model consisting of adding noise at every pre- and post-activation in the network, arguing that the resulting posterior can be sampled using an efficient Gibbs sampler. For small models, the Gibbs sampler attains similar performances as the state-of-the-art Markov chain Monte Carlo (MCMC) methods, such as the Hamiltonian Monte Carlo (HMC) or the Metropolis adjusted Langevin algorithm (MALA), both on real and synthetic data. By framing our analysis in the teacher-student setting, we introduce a thermalization criterion that allows us to detect when an algorithm, when run on data with synthetic labels, fails to sample from the posterior. The criterion is based on the fact that in the teacher-student setting we can initialize an algorithm directly at equilibrium.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 61 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

26 of 61 references.

References · 26

  1. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  2. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  3. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  4. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  5. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 336
  6. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  7. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  8. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  9. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  10. 2020Bayesian Neural Networks: An Introduction and Surveycited by 130
  11. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  12. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
14 more
  1. 2017Measuring Sample Quality with Kernelscited by 119
  2. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  3. 2017TensorFlow Distributionscited by 105
  4. 2015Measuring Sample Quality with Stein's Methodcited by 94
  5. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 74
  6. 2018Bayesian Uncertainty Estimation for Batch Normalized Deep Networkscited by 70
  7. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  8. 2019Stochastic gradient Markov chain Monte Carlocited by 56
  9. 2020Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splittingcited by 54
  10. 2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
  11. 2016Natural-Parameter Networks: A Class of Probabilistic Neural Networkscited by 33
  12. 2022Bayesian Learning for Neural Networks: an algorithmic surveycited by 22
  13. 2022A Theoretical Framework for Inference Learningcited by 17
  14. 2014Techniques for Learning Binary Stochastic Feedforward Neural Networkscited by 14

Built on by

Not cited yet.