mlsanityData to 7 Oct 2026

Low-Precision Stochastic Gradient Langevin Dynamics

arXiv 2206.09909 ↗First posted 20 Jun 2022cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored. As a consequence, sampling is simply infeasible in many large-scale scenarios, despite providing remarkable benefits to generalization and uncertainty estimation for neural networks. In this paper, we provide the first study of low-precision Stochastic Gradient Langevin Dynamics (SGLD), showing that its costs can be significantly reduced without sacrificing performance, due to its intrinsic ability to handle system noise. We prove that the convergence of low-precision SGLD with full-precision gradient accumulators is less affected by the quantization error than its SGD counterpart in the strongly convex setting. To further enable low-precision gradient accumulators, we develop a new quantization function for SGLD that preserves the variance in each update step. We demonstrate that low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits on a variety of deep learning tasks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

36 of 54 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2019Learned Step Size Quantizationcited by 604
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  7. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  8. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  9. 2015Deep Learning with Limited Numerical Precisioncited by 368
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  11. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  12. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
24 more
  1. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  2. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  3. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  4. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  5. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  6. 2017Training Quantized Nets: A Deeper Understandingcited by 74
  7. 2018Sampling Can Be Faster Than Optimizationcited by 68
  8. 2015Bayesian Dark Knowledgecited by 45
  9. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 40
  10. 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 38
  11. 2018Adversarial Distillation of Bayesian Neural Network Posteriorscited by 27
  12. 2020A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributionscited by 26
  13. 2017Control Variates for Stochastic Gradient MCMCcited by 25
  14. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 22
  15. 2019QPyTorch: A Low-Precision Arithmetic Simulation Frameworkcited by 21
  16. 2018VIBNN: Hardware Acceleration of Bayesian Neural Networkscited by 19
  17. 2019Bayesian Inference for Large Scale Image Classificationcited by 18
  18. 2016Stochastic Gradient MCMC with Stale Gradientscited by 9
  19. 2016Scalable Bayesian Learning of Recurrent Neural Networks for Language Modelingcited by 9
  20. 2021On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networkscited by 8
  21. 2017Computation Error Analysis of Block Floating Point Arithmetic Oriented Convolution Neural Network Accelerator Designcited by 7
  22. 2019Computational Separations between Sampling and Optimizationcited by 7
  23. 2015Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagationcited by 6
  24. 2019Sampling-Free Learning of Bayesian Quantized Neural Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 4; 2025: 2.

12 papers.

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  3. 2023Fortuna: A Library for Uncertainty Quantification in Deep Learningcited by 14
  4. 2023On the Convergence of Stochastic Gradient Descent in Low-precision Number Formatscited by 4
  5. 2024BOLD: Boolean Logic Deep Learningcited by 3
  6. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  7. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  8. 2023Mathematical Challenges in Deep Learningcited by 0
  9. 2023Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolutioncited by 0
  10. 2023Be Bayesian by Attachments to Catch More Uncertaintycited by 0
  11. 2024Boolean Logic as an Error feedback mechanismcited by 0
  12. 2025Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?cited by 0