# FPRaker: A Processing Element For Accelerating Neural Network Training

arXiv: https://arxiv.org/abs/2010.08065 · HTML: https://mlsanity.com/paper/2010.08065/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration](https://mlsanity.com/paper/2411.11745/) (2024)
- [Demystifying BERT: Implications for Accelerator Design](https://mlsanity.com/paper/2104.08335/) (2021)
- [EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Accelerators](https://mlsanity.com/paper/2202.02310/) (2022)
- [Schr"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training](https://mlsanity.com/paper/2204.13666/) (2022)
- [MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration](https://mlsanity.com/paper/2609.19683/) (2026)

## Abstract (arXiv)

We present FPRaker, a processing element for composing training accelerators. FPRaker processes several floating-point multiply-accumulation operations concurrently and accumulates their result into a higher precision accumulator. FPRaker boosts performance and energy efficiency during training by taking advantage of the values that naturally appear during training. Specifically, it processes the significand of the operands of each multiply-accumulate as a series of signed powers of two. The conversion to this form is done on-the-fly. This exposes ineffectual work that can be skipped: values when encoded have few terms and some of them can be discarded as they would fall outside the range of the accumulator given the limited precision of floating-point. We demonstrate that FPRaker can be used to compose an accelerator for training and that it can improve performance and energy efficiency compared to using conventional floating-point units under ISO-compute area constraints. We also demonstrate that FPRaker delivers additional benefits when training incorporates pruning and quantization. Finally, we show that FPRaker naturally amplifies performance with training methods that use a different precision per layer.