# FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding

arXiv: https://arxiv.org/abs/2110.15456 · HTML: https://mlsanity.com/paper/2110.15456/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/1711.02213/) (2017)
- [Training DNNs with Hybrid Block Floating Point](https://mlsanity.com/paper/1804.01526/) (2018)
- [Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training](https://mlsanity.com/paper/2009.10976/) (2020)
- [TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference](https://mlsanity.com/paper/2009.00748/) (2020)
- [Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization](https://mlsanity.com/paper/1811.04770/) (2018)
- [Low Precision RNNs: Quantizing RNNs Without Losing Accuracy](https://mlsanity.com/paper/1710.07706/) (2017)

## Built on by (most cited first, 30 of 31)

- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [LQER: Low-Rank Quantization Error Reconstruction for LLMs](https://mlsanity.com/paper/2402.02446/) (2024)
- [Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization](https://mlsanity.com/paper/2406.12930/) (2024)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning](https://mlsanity.com/paper/2502.12913/) (2025)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [Effective Interplay between Sparsity and Quantization: From Theory to Practice](https://mlsanity.com/paper/2405.20935/) (2024)
- [HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization](https://mlsanity.com/paper/2405.19751/) (2024)
- [MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving](https://mlsanity.com/paper/2510.14557/) (2025)
- [FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Support](https://mlsanity.com/paper/2203.06673/) (2022)
- [Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format](https://mlsanity.com/paper/2411.15982/) (2024)
- [DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics](https://mlsanity.com/paper/2403.14353/) (2024)
- [Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving](https://mlsanity.com/paper/2507.10178/) (2025)
- [Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design](https://mlsanity.com/paper/2309.13015/) (2023)
- [Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing](https://mlsanity.com/paper/2510.16040/) (2025)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [On Stochastic Rounding with Few Random Bits](https://mlsanity.com/paper/2504.20634/) (2025)
- [Cut Less, Fold More: Model Compression through the Lens of Projection Geometry](https://mlsanity.com/paper/2602.18116/) (2026)
- [Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training](https://mlsanity.com/paper/2211.10737/) (2022)
- [CAMEL: Co-Designing AI Models and Embedded DRAMs for Efficient On-Device Learning](https://mlsanity.com/paper/2305.03148/) (2023)
- [Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization](https://mlsanity.com/paper/2501.06663/) (2025)
- [Low-Precision Streaming PCA](https://mlsanity.com/paper/2510.22440/) (2025)
- [Search Your Block Floating Point Scales!](https://mlsanity.com/paper/2605.12464/) (2026)
- [Mirage: An RNS-Based Photonic Accelerator for DNN Training](https://mlsanity.com/paper/2311.17323/) (2023)
- [Schr"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training](https://mlsanity.com/paper/2204.13666/) (2022)
- [DAISM: Digital Approximate In-SRAM Multiplier-based Accelerator for DNN Training and Inference](https://mlsanity.com/paper/2305.07376/) (2023)
- [BIM: Block-Wise Self-Supervised Learning with Masked Image Modeling](https://mlsanity.com/paper/2311.17218/) (2023)
- [BitQ: Tailoring Block Floating Point Precision for Improved DNN Efficiency on Resource-Constrained Devices](https://mlsanity.com/paper/2409.17093/) (2024)
- [Distribution-Aware Hadamard Quantization for Hardware-Efficient Implicit Neural Representations](https://mlsanity.com/paper/2508.13478/) (2025)
- [MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference](https://mlsanity.com/paper/2607.17733/) (2026)

## Abstract (arXiv)

Block Floating Point (BFP) can efficiently support quantization for Deep Neural Network (DNN) training by providing a wide dynamic range via a shared exponent across a group of values. In this paper, we propose a Fast First, Accurate Second Training (FAST) system for DNNs, where the weights, activations, and gradients are represented in BFP. FAST supports matrix multiplication with variable precision BFP input operands, enabling incremental increases in DNN precision throughout training. By increasing the BFP precision across both training iterations and DNN layers, FAST can greatly shorten the training time while reducing overall hardware resource usage. Our FAST Multipler-Accumulator (fMAC) supports dot product computations under multiple BFP precisions. We validate our FAST system on multiple DNNs with different datasets, demonstrating a 2-6× speedup in training on a single-chip platform over prior work based on mixed-precision or block floating point number systems while achieving similar performance in validation accuracy.