# A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training

arXiv: https://arxiv.org/abs/2404.14010 · HTML: https://mlsanity.com/paper/2404.14010/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2024.

## Built on by (most cited first, 6 of 6)

- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Automating Hardware Design and Verification from Architectural Papers via a Neural-Symbolic Graph Framework](https://mlsanity.com/paper/2511.06067/) (2025)
- [Stochastic Rounding Increases Small Singular Values](https://mlsanity.com/paper/2606.00312/) (2026)
- [SEADA: An efficient methodology for optimizing mixed-precision DNNs on multi-precision spatial architectures](https://mlsanity.com/paper/2606.27884/) (2026)
- [MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference](https://mlsanity.com/paper/2609.28358/) (2026)
- [Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2](https://mlsanity.com/paper/2610.01889/) (2026)

## Abstract (arXiv)

Training Deep Neural Networks (DNNs) can be computationally demanding, particularly when dealing with large models. Recent work has aimed to mitigate this computational challenge by introducing 8-bit floating-point (FP8) formats for multiplication. However, accumulations are still done in either half (16-bit) or single (32-bit) precision arithmetic. In this paper, we investigate lowering accumulator word length while maintaining the same model accuracy. We present a multiply-accumulate (MAC) unit with FP8 multiplier inputs and FP12 accumulations, which leverages an optimized stochastic rounding (SR) implementation to mitigate swamping errors that commonly arise during low precision accumulations. We investigate the hardware implications and accuracy impact associated with varying the number of random bits used for rounding operations. We additionally attempt to reduce MAC area and power by proposing a new scheme to support SR in floating-point MAC and by removing support for subnormal values. Our optimized eager SR unit significantly reduces delay and area when compared to a classic lazy SR design. Moreover, when compared to MACs utilizing single-or half-precision adders, our design showcases notable savings in all metrics. Furthermore, our approach consistently maintains near baseline accuracy across a diverse range of computer vision tasks, making it a promising alternative for low-precision DNN training.