mlsanityData to 14 Oct 2026

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

arXiv 1902.03932 ↗First posted 11 Feb 2019cs.LG cs.AI cs.CVcited by 203

Not on the map. Why?

Abstract (arXiv)

The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We also prove non-asymptotic convergence of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the scalability and effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 44; 2022: 46; 2023: 41; 2024: 38; 2025: 28; 2026: 6.

203 papers.

Sort:

By area:

  1. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  2. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  3. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  4. 2021Bayesian Neural Network Priors Revisitedcited by 121
  5. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  6. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 104
  7. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  8. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  9. 2021Learning Neural Network Subspacescited by 82
  10. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  11. 2023LoRA ensembles for large language model fine-tuningcited by 67
  12. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
38 more
  1. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  2. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  3. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  4. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  5. 2022A Langevin-like Sampler for Discrete Distributionscited by 44
  6. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  7. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 37
  8. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  9. 2023BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inferencecited by 35
  10. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  11. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  12. 2021Diversity and Generalization in Neural Network Ensemblescited by 32
  13. 2022Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reductioncited by 32
  14. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  15. 2021Exact Langevin Dynamics with Stochastic Gradientscited by 30
  16. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 28
  17. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  18. 2023Introducing an Improved Information-Theoretic Measure of Predictive Uncertaintycited by 28
  19. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  20. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  21. 2021Annealed Stein Variational Gradient Descentcited by 26
  22. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  23. 2022Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgottencited by 25
  24. 2022Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEscited by 25
  25. 2021A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applicationscited by 24
  26. 2022Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constantscited by 24
  27. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  28. 2021On Stein Variational Neural Network Ensemblescited by 21
  29. 2022Knowledge Removal in Sampling-based Bayesian Inferencecited by 21
  30. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  31. 2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
  32. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  33. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  34. 2021Bayesian OOD detection with aleatoric uncertainty and outlier exposurecited by 16
  35. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  36. 2021When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecastingcited by 16
  37. 2024On Information-Theoretic Measures of Predictive Uncertaintycited by 16
  38. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16

50 most cited of 203 All 203 (JSON)