# On Stochastic Rounding with Few Random Bits

arXiv: https://arxiv.org/abs/2504.20634 · HTML: https://mlsanity.com/paper/2504.20634/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 15 references that are in the map.
- First posted on arXiv 29 Apr 2025.

## Builds on (most cited first, 3 of 3)

- [FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding](https://mlsanity.com/paper/2110.15456/) (2021)
- [8-bit Numerical Formats for Deep Neural Networks](https://mlsanity.com/paper/2206.02915/) (2022)
- [Improved stochastic rounding](https://mlsanity.com/paper/2006.00489/) (2020)

## Built on by (most cited first, 4 of 4)

- [Elucidating the Design Space of FP4 training](https://mlsanity.com/paper/2509.17791/) (2025)
- [UE5M3 FP4 Block Scaling for Stable Language Model Pretraining](https://mlsanity.com/paper/2609.02846/) (2026)
- [Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learning](https://mlsanity.com/paper/2606.04028/) (2026)
- [Format-Aware Fusion for Fast FP4 Pretraining](https://mlsanity.com/paper/2610.00053/) (2026)

## Abstract (arXiv)

Large-scale numerical computations make increasing use of low-precision (LP) floating point formats and mixed precision arithmetic, which can be enhanced by the technique of stochastic rounding (SR), that is, rounding an intermediate high-precision value up or down randomly as a function of the value's distance to the two rounding candidates. Stochastic rounding requires, in addition to the high-precision input value, a source of random bits. As the provision of high-quality random bits is an additional computational cost, it is of interest to require as few bits as possible while maintaining the desirable properties of SR in a given computation, or computational domain. This paper examines a number of possible implementations of few-bit stochastic rounding (FBSR), and shows how several natural implementations can introduce sometimes significant bias into the rounding process, which are not present in the case of infinite-bit, infinite-precision examinations of these implementations. The paper explores the impact of these biases in machine learning examples, and hence opens another class of configuration parameters of which practitioners should be aware when developing or adopting low-precision floating point. Code is available at http://github.com/graphcore-research/arith25-stochastic-rounding.