mlsanityData to 7 Oct 2026

Stochastic Gradient Descent as Approximate Bayesian Inference

arXiv 1704.04289 ↗First posted 13 Apr 2017stat.ML cs.LGcited by 298

Not on the map. Why?

Abstract (arXiv)

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior, minimizing the Kullback-Leibler divergence between these two distributions. (2) We demonstrate that constant SGD gives rise to a new variational EM algorithm that optimizes hyperparameters in complex probabilistic models. (3) We also propose SGD with momentum for sampling and show how to adjust the damping coefficient accordingly. (4) We analyze MCMC algorithms. For Langevin Dynamics and Stochastic Gradient Fisher Scoring, we quantify the approximation errors due to finite learning rates. Finally (5), we use the stochastic process perspective to give a short proof of why Polyak averaging is optimal. Based on this idea, we propose a scalable approximate MCMC algorithm, the Averaged Stochastic Gradient Sampler.

Key facts. Cited by 298 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 38; 2023: 47; 2024: 29; 2025: 60; 2026: 72.

298 papers.

Sort:

By area:

  1. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  2. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  3. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  4. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  5. 2021Epistemic Neural Networkscited by 120
  6. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  7. 2021The Bayesian Learning Rulecited by 95
  8. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  9. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  10. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  11. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  12. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
38 more
  1. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  2. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  3. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  4. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  5. 2021Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networkscited by 43
  6. 2021LiBRe: A Practical Bayesian Approach to Adversarial Detectioncited by 40
  7. 2022SAM as an Optimal Relaxation of Bayescited by 40
  8. 2021Strength of Minibatch Noise in SGDcited by 39
  9. 2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
  10. 2021Differentiable Annealed Importance Sampling and the Perils of Gradient Noisecited by 36
  11. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  12. 2021Batch size-invariance for policy optimizationcited by 33
  13. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraintscited by 33
  14. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  15. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  16. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  17. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  18. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  19. 2021Properties of the After Kernelcited by 27
  20. 2023Sampling from Gaussian Process Posteriors using Stochastic Gradient Descentcited by 27
  21. 2022NeuralEF: Deconstructing Kernels by Deep Neural Networkscited by 26
  22. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  23. 2021An Effective Baseline for Robustness to Distributional Shiftcited by 25
  24. 2021Sqrt(d) Dimension Dependence of Langevin Monte Carlocited by 25
  25. 2021A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applicationscited by 24
  26. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  27. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  28. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  29. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  30. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20
  31. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 20
  32. 2025Generative Uncertainty in Diffusion Modelscited by 20
  33. 2022Efficient Bayesian Uncertainty Estimation for nnU-Netcited by 19
  34. 2023Practical Differentially Private Hyperparameter Tuning with Subsamplingcited by 19
  35. 2023Federated Learning as Variational Inference: A Scalable Expectation Propagation Approachcited by 17
  36. 2021A Bayesian neural network predicts the dissolution of compact planetary systemscited by 15
  37. 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
  38. 2021Assessments of epistemic uncertainty using Gaussian stochastic weight averaging for fluid-flow regressioncited by 15

50 most cited of 298 All 298 (JSON)