# Low-Precision Stochastic Gradient Langevin Dynamics

arXiv: https://arxiv.org/abs/2206.09909 · HTML: https://mlsanity.com/paper/2206.09909/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 54 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [Fixed Point Quantization of Deep Convolutional Networks](https://mlsanity.com/paper/1511.06393/) (2015)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient](https://mlsanity.com/paper/1710.00095/) (2017)
- [Training and Inference with Integers in Deep Neural Networks](https://mlsanity.com/paper/1802.04680/) (2018)
- [Training Quantized Nets: A Deeper Understanding](https://mlsanity.com/paper/1706.02379/) (2017)
- [Sampling Can Be Faster Than Optimization](https://mlsanity.com/paper/1811.08413/) (2018)
- [Bayesian Dark Knowledge](https://mlsanity.com/paper/1506.04416/) (2015)
- [SWALP : Stochastic Weight Averaging in Low-Precision Training](https://mlsanity.com/paper/1904.11943/) (2019)
- [Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?](https://mlsanity.com/paper/1806.07550/) (2018)
- [Adversarial Distillation of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/1806.10317/) (2018)
- [A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions](https://mlsanity.com/paper/2010.09800/) (2020)
- [Control Variates for Stochastic Gradient MCMC](https://mlsanity.com/paper/1706.05439/) (2017)
- [Training Binary Neural Networks using the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10778/) (2020)
- [QPyTorch: A Low-Precision Arithmetic Simulation Framework](https://mlsanity.com/paper/1910.04540/) (2019)
- [VIBNN: Hardware Acceleration of Bayesian Neural Networks](https://mlsanity.com/paper/1802.00822/) (2018)
- [Bayesian Inference for Large Scale Image Classification](https://mlsanity.com/paper/1908.03491/) (2019)
- [Stochastic Gradient MCMC with Stale Gradients](https://mlsanity.com/paper/1610.06664/) (2016)

## Built on by (most cited first, 12 of 12)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Quantification of Uncertainty with Adversarial Models](https://mlsanity.com/paper/2307.03217/) (2023)
- [Fortuna: A Library for Uncertainty Quantification in Deep Learning](https://mlsanity.com/paper/2302.04019/) (2023)
- [On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats](https://mlsanity.com/paper/2301.01651/) (2023)
- [BOLD: Boolean Logic Deep Learning](https://mlsanity.com/paper/2405.16339/) (2024)
- [Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks](https://mlsanity.com/paper/2506.17768/) (2025)
- [Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems](https://mlsanity.com/paper/2411.14860/) (2024)
- [Mathematical Challenges in Deep Learning](https://mlsanity.com/paper/2303.15464/) (2023)
- [Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolution](https://mlsanity.com/paper/2305.18864/) (2023)
- [Be Bayesian by Attachments to Catch More Uncertainty](https://mlsanity.com/paper/2310.13027/) (2023)
- [Boolean Logic as an Error feedback mechanism](https://mlsanity.com/paper/2401.16418/) (2024)
- [Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?](https://mlsanity.com/paper/2508.07142/) (2025)

## Abstract (arXiv)

While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored. As a consequence, sampling is simply infeasible in many large-scale scenarios, despite providing remarkable benefits to generalization and uncertainty estimation for neural networks. In this paper, we provide the first study of low-precision Stochastic Gradient Langevin Dynamics (SGLD), showing that its costs can be significantly reduced without sacrificing performance, due to its intrinsic ability to handle system noise. We prove that the convergence of low-precision SGLD with full-precision gradient accumulators is less affected by the quantization error than its SGD counterpart in the strongly convex setting. To further enable low-precision gradient accumulators, we develop a new quantization function for SGLD that preserves the variance in each update step. We demonstrate that low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits on a variety of deep learning tasks.