# On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation

arXiv: https://arxiv.org/abs/2202.12276 · HTML: https://mlsanity.com/paper/2202.12276/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 37 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 6 of 6)

- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Training Quantized Nets: A Deeper Understanding](https://mlsanity.com/paper/1706.02379/) (2017)
- [Stochastic rounding and reduced-precision fixed-point arithmetic for solving neural ordinary differential equations](https://mlsanity.com/paper/1904.11263/) (2019)
- [Low-Precision Floating-Point Schemes for Neural Network Training](https://mlsanity.com/paper/1804.05267/) (2018)
- [Stochastic Rounding: Algorithms and Hardware Accelerator](https://mlsanity.com/paper/2001.01501/) (2020)

## Built on by (most cited first, 4 of 4)

- [Stochastic Rounding for LLM Training: Theory and Practice](https://mlsanity.com/paper/2502.20566/) (2025)
- [Low-Precision Streaming PCA](https://mlsanity.com/paper/2510.22440/) (2025)
- [Accelerated Gradient Methods with Biased Gradient Estimates: Risk Sensitivity, High-Probability Guarantees, and Large Deviation Bounds](https://mlsanity.com/paper/2509.13628/) (2025)
- [Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding](https://mlsanity.com/paper/2511.00874/) (2025)

## Abstract (arXiv)

When implementing the gradient descent method in low precision, the employment of stochastic rounding schemes helps to prevent stagnation of convergence caused by the vanishing gradient effect. Unbiased stochastic rounding yields zero bias by preserving small updates with probabilities proportional to their relative magnitudes. This study provides a theoretical explanation for the stagnation of the gradient descent method in low-precision computation. Additionally, we propose two new stochastic rounding schemes that trade the zero bias property with a larger probability to preserve small gradients. Our methods yield a constant rounding bias that, on average, lies in a descent direction. For convex problems, we prove that the proposed rounding methods typically have a beneficial effect on the convergence rate of gradient descent. We validate our theoretical analysis by comparing the performances of various rounding schemes when optimizing a multinomial logistic regression model and when training a simple neural network with an 8-bit floating-point format.