mlsanityData to 14 Oct 2026

Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?

arXiv 2508.07142 ↗First posted 10 Aug 2025cs.LG cs.AI cs.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Low-precision training has become crucial for reducing the computational and memory costs of large-scale deep learning. However, quantizing gradients introduces magnitude shrinkage, which can change how stochastic gradient descent (SGD) converges. In this study, we explore SGD convergence under a gradient shrinkage model, where each stochastic gradient is scaled by a factor ( qk ∈ (0,1] ). We show that this shrinkage affect the usual stepsize ( μk ) with an effective stepsize ( μk qk ), slowing convergence when ( qmin < 1 ). With typical smoothness and bounded-variance assumptions, we prove that low-precision SGD still converges, but at a slower pace set by ( qmin ), and with a higher steady error level due to quantization effects. We analyze theoretically how lower numerical precision slows training by treating it as gradient shrinkage within the standard SGD convergence setup.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 10 Aug 2025.

Builds on

22 of 31 references.

References · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  4. 2017Mixed Precision Trainingcited by 1,156
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  7. 2019The State of Sparsity in Deep Neural Networkscited by 476
  8. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  9. 2022FP8 Formats for Deep Learningcited by 196
  10. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  11. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 143
  12. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
10 more
  1. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 78
  2. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  3. 2019Fighting Quantization Bias With Biascited by 30
  4. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
  5. 2018High-Accuracy Low-Precision Trainingcited by 20
  6. 2022Low-Precision Stochastic Gradient Langevin Dynamicscited by 12
  7. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  8. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  9. 2023On the Convergence of Stochastic Gradient Descent in Low-precision Number Formatscited by 4
  10. 2023Revisiting 16-bit Neural Network Training: A Practical Approach for Resource-Limited Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
  2. 2022Flatter, faster: scaling momentum for optimal speedup of SGD
  3. 2026Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models
  4. 2025Can SGD Handle Heavy-Tailed Noise?
  5. 2023Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian Monte Carlo
  6. 2024Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees
  7. 2023On the different regimes of Stochastic Gradient Descent
  8. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks