mlsanityData to 7 Oct 2026

Structured Stochastic Gradient MCMC

arXiv 2107.09028 ↗First posted 19 Jul 2021cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient Markov Chain Monte Carlo (SGMCMC) is considered the gold standard for Bayesian inference in large-scale models, such as Bayesian neural networks. Since practitioners face speed versus accuracy tradeoffs in these models, variational inference (VI) is often the preferable option. Unfortunately, VI makes strong assumptions on both the factorization and functional form of the posterior. In this work, we propose a new non-parametric variational approximation that makes no assumptions about the approximate posterior's functional form and allows practitioners to specify the exact dependencies the algorithm should respect or break. The approach relies on a new Langevin-type algorithm that operates on a modified energy function, where parts of the latent variables are averaged over samples from earlier iterations of the Markov chain. This way, statistical dependencies can be broken in a controlled way, allowing the chain to mix faster. This scheme can be further modified in a "dropout" manner, leading to even more scalability. We test our scheme for ResNet-20 on CIFAR-10, SVHN, and FMNIST. In all cases, we find improvements in convergence speed and/or final accuracy compared to SG-MCMC and VI.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 90 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

32 of 90 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2016Variational Inference: A Review for Statisticianscited by 1,466
  7. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  8. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  9. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  11. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 336
  12. 2013Black Box Variational Inferencecited by 317
20 more
  1. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  2. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  3. 2017Advances in Variational Inferencecited by 236
  4. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  5. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  6. 2017TensorFlow Distributionscited by 105
  7. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  8. 2017Generalizing Hamiltonian Monte Carlo with Neural Networkscited by 52
  9. 2019Statistical Guarantees for the Robustness of Bayesian Neural Networkscited by 35
  10. 2015Early Stopping is Nonparametric Variational Inferencecited by 23
  11. 2014Structured Stochastic Variational Inferencecited by 21
  12. 2020Automatic structured variational inferencecited by 21
  13. 2019Bayesian Inference for Large Scale Image Classificationcited by 18
  14. 2018Meta-Learning for Stochastic Gradient MCMCcited by 16
  15. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  16. 2015Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayescited by 14
  17. 2019Partitioned integrators for thermodynamic parameterization of neural networkscited by 9
  18. 2017Stochastic Gradient MCMC Methods for Hidden Markov Modelscited by 8
  19. 2013Variational MCMCcited by 7
  20. 2017A Divergence Bound for Hybrids of MCMC and Variational Inference and an Application to Langevin Dynamics and SGVIcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 5; 2025: 1; 2026: 2.

12 papers.

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2024Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inferencecited by 16
  3. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 15
  4. 2024Scalable Bayesian Learning with posteriorscited by 13
  5. 2022Approximate blocked Gibbs sampling for Bayesian neural networkscited by 3
  6. 2024Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Samplingcited by 3
  7. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  8. 2022A PAC-Bayes oracle inequality for sparse neural networkscited by 2
  9. 2024Functional Stochastic Gradient MCMC for Bayesian Neural Networkscited by 2
  10. 2023The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networkscited by 0
  11. 2026Neural-Inspired Posterior Approximation (NIPA)cited by 0
  12. 2026Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlocited by 0