# A Simple and Efficient Stochastic Rounding Method for Training Neural Networks in Low Precision

arXiv: https://arxiv.org/abs/2103.13445 · HTML: https://mlsanity.com/paper/2103.13445/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 5 of 5)

- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Low-Precision Floating-Point Schemes for Neural Network Training](https://mlsanity.com/paper/1804.05267/) (2018)
- [Stochastic Rounding: Algorithms and Hardware Accelerator](https://mlsanity.com/paper/2001.01501/) (2020)

## Built on by (most cited first, 3 of 3)

- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [any4: Learned 4-bit Numeric Representation for LLMs](https://mlsanity.com/paper/2507.04610/) (2025)
- [Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding](https://mlsanity.com/paper/2511.00874/) (2025)

## Abstract (arXiv)

Conventional stochastic rounding (CSR) is widely employed in the training of neural networks (NNs), showing promising training results even in low-precision computations. We introduce an improved stochastic rounding method, that is simple and efficient. The proposed method succeeds in training NNs with 16-bit fixed-point numbers and provides faster convergence and higher classification accuracy than both CSR and deterministic rounding-to-the-nearest method.