mlsanityData to 7 Oct 2026

Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding

arXiv 2511.00874 ↗First posted 2 Nov 2025cs.LG cs.NA math.NAcited by 1

Not on the map. Why?

Abstract (arXiv)

LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors -- especially batch size -- remains under explored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during back-propagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

23 of 33 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016Wide Residual Networkscited by 3,057
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2019Learned Step Size Quantizationcited by 604
  7. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  8. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  9. 2015Deep Learning with Limited Numerical Precisioncited by 368
  10. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  11. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  12. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
11 more
  1. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  2. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  3. 2017Training Quantized Nets: A Deeper Understandingcited by 74
  4. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  5. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  6. 2019Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integerscited by 32
  7. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  8. 2022On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computationcited by 4
  9. 2023On the Convergence of the Gradient Descent Method with Stochastic Fixed-point Rounding Errors under the Polyak-Lojasiewicz Inequalitycited by 4
  10. 2021A Simple and Efficient Stochastic Rounding Method for Training Neural Networks in Low Precisioncited by 3
  11. 2023Training with Mixed-Precision Floating-Point Assignmentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Auditing Invisible Weight Updates with Reference Tracescited by 0