# Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian Monte Carlo

arXiv: https://arxiv.org/abs/2310.16320 · HTML: https://mlsanity.com/paper/2310.16320/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 31 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 5 of 5)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Training and Inference with Integers in Deep Neural Networks](https://mlsanity.com/paper/1802.04680/) (2018)
- [Stable and low-precision training for large-scale vision-language models](https://mlsanity.com/paper/2304.13013/) (2023)

## Built on by (most cited first, 1 of 1)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)

## Abstract (arXiv)

Low-precision training has emerged as a promising low-cost technique to enhance the training efficiency of deep neural networks without sacrificing much accuracy. Its Bayesian counterpart can further provide uncertainty quantification and improved generalization accuracy. This paper investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) with low-precision and full-precision gradient accumulators for both strongly log-concave and non-log-concave distributions. Theoretically, our results show that, to achieve ε-error in the 2-Wasserstein distance for non-log-concave distributions, low-precision SGHMC achieves quadratic improvement (widetildeOleft(ε-2μ^*-2log2left(ε-1right)right)) compared to the state-of-the-art low-precision sampler, Stochastic Gradient Langevin Dynamics (SGLD) (widetildeOleft(ε-4λ*-1log5left(ε-1right)right)). Moreover, we prove that low-precision SGHMC is more robust to the quantization error compared to low-precision SGLD due to the robustness of the momentum-based update w.r.t. gradient noise. Empirically, we conduct experiments on synthetic data, and MNIST, CIFAR-10 & CIFAR-100 datasets, which validate our theoretical findings. Our study highlights the potential of low-precision SGHMC as an efficient and accurate sampling method for large-scale and resource-limited machine learning.