# On the Convergence of the Gradient Descent Method with Stochastic Fixed-point Rounding Errors under the Polyak-Lojasiewicz Inequality

arXiv: https://arxiv.org/abs/2301.09511 · HTML: https://mlsanity.com/paper/2301.09511/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 23 Jan 2023.

## Builds on (most cited first, 11 of 11)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Stability and Generalization of Learning Algorithms that Converge to Global Optima](https://mlsanity.com/paper/1710.08402/) (2017)
- [Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology](https://mlsanity.com/paper/2002.07867/) (2020)
- [Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs](https://mlsanity.com/paper/2103.10873/) (2021)
- [Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent](https://mlsanity.com/paper/2106.13792/) (2021)
- [Low-Precision Floating-Point Schemes for Neural Network Training](https://mlsanity.com/paper/1804.05267/) (2018)
- [When Does Stochastic Gradient Algorithm Work Well?](https://mlsanity.com/paper/1801.06159/) (2018)

## Built on by (most cited first, 4 of 4)

- [AdamL: A fast adaptive gradient method incorporating loss function](https://mlsanity.com/paper/2312.15295/) (2023)
- [Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding](https://mlsanity.com/paper/2511.00874/) (2025)
- [Stochastic Rounding Increases Small Singular Values](https://mlsanity.com/paper/2606.00312/) (2026)
- [Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameters](https://mlsanity.com/paper/2606.06934/) (2026)

## Abstract (arXiv)

When training neural networks with low-precision computation, rounding errors often cause stagnation or are detrimental to the convergence of the optimizers; in this paper we study the influence of rounding errors on the convergence of the gradient descent method for problems satisfying the Polyak-Lojasiewicz inequality. Within this context, we show that, in contrast, biased stochastic rounding errors may be beneficial since choosing a proper rounding strategy eliminates the vanishing gradient problem and forces the rounding bias in a descent direction. Furthermore, we obtain a bound on the convergence rate that is stricter than the one achieved by unbiased stochastic rounding. The theoretical analysis is validated by comparing the performances of various rounding strategies when optimizing several examples using low-precision fixed-point number formats.