# High-Accuracy Low-Precision Training

arXiv: https://arxiv.org/abs/1803.03383 · HTML: https://mlsanity.com/paper/1803.03383/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2018.

## Built on by (most cited first, 20 of 20)

- [Pufferfish: Communication-efficient Models At No Extra Cost](https://mlsanity.com/paper/2103.03936/) (2021)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning](https://mlsanity.com/paper/2302.11147/) (2023)
- [Collage: Light-Weight Low-Precision Strategy for LLM Training](https://mlsanity.com/paper/2405.03637/) (2024)
- [Guaranteed Approximation Bounds for Mixed-Precision Neural Operators](https://mlsanity.com/paper/2307.15034/) (2023)
- [Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale](https://mlsanity.com/paper/2105.12676/) (2021)
- [LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update](https://mlsanity.com/paper/2106.13914/) (2021)
- [Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators](https://mlsanity.com/paper/2101.11748/) (2021)
- [MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point](https://mlsanity.com/paper/2207.08867/) (2022)
- [BOLD: Boolean Logic Deep Learning](https://mlsanity.com/paper/2405.16339/) (2024)
- [Low-Precision Streaming PCA](https://mlsanity.com/paper/2510.22440/) (2025)
- [How Low Can We Go: Trading Memory for Error in Low-Precision Training](https://mlsanity.com/paper/2106.09686/) (2021)
- [Training with Mixed-Precision Floating-Point Assignments](https://mlsanity.com/paper/2301.13464/) (2023)
- [Noisy Truncated SGD: Optimization and Generalization](https://mlsanity.com/paper/2103.00075/) (2021)
- [Safe Beyond the Horizon: Efficient Sampling-based MPC with Neural Control Barrier Functions](https://mlsanity.com/paper/2502.15006/) (2025)
- [KML: Using Machine Learning to Improve Storage Systems](https://mlsanity.com/paper/2111.11554/) (2021)
- [Boolean Logic as an Error feedback mechanism](https://mlsanity.com/paper/2401.16418/) (2024)
- [Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?](https://mlsanity.com/paper/2508.07142/) (2025)
- [SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions](https://mlsanity.com/paper/2510.08999/) (2025)

## Abstract (arXiv)

Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it. Still, it has been used primarily for inference - not training. Previous low-precision training algorithms suffered from a fundamental tradeoff: as the number of bits of precision is lowered, quantization noise is added to the model, which limits statistical accuracy. To address this issue, we describe a simple low-precision stochastic gradient descent variant called HALP. HALP converges at the same theoretical rate as full-precision algorithms despite the noise introduced by using low precision throughout execution. The key idea is to use SVRG to reduce gradient variance, and to combine this with a novel technique called bit centering to reduce quantization error. We show that on the CPU, HALP can run up to 4 × faster than full-precision SVRG and can match its convergence trajectory. We implemented HALP in TensorQuant, and show that it exceeds the validation performance of plain low-precision SGD on two deep learning tasks.