mlsanityData to 14 Oct 2026

FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding

arXiv 2110.15456 ↗First posted 28 Oct 2021cs.LG cs.ARcited by 31

Not on the map. Why?

Abstract (arXiv)

Block Floating Point (BFP) can efficiently support quantization for Deep Neural Network (DNN) training by providing a wide dynamic range via a shared exponent across a group of values. In this paper, we propose a Fast First, Accurate Second Training (FAST) system for DNNs, where the weights, activations, and gradients are represented in BFP. FAST supports matrix multiplication with variable precision BFP input operands, enabling incremental increases in DNN precision throughout training. By increasing the BFP precision across both training iterations and DNN layers, FAST can greatly shorten the training time while reducing overall hardware resource usage. Our FAST Multipler-Accumulator (fMAC) supports dot product computations under multiple BFP precisions. We validate our FAST system on multiple DNNs with different datasets, demonstrating a 2-6× speedup in training on a single-chip platform over prior work based on mixed-precision or block floating point number systems while achieving similar performance in validation accuracy.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 16 of its 46 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

16 of 46 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2016YOLO9000: Better, Faster, Strongercited by 1,431
  4. 2017Mixed Precision Trainingcited by 1,156
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  7. 2015Deep Learning with Limited Numerical Precisioncited by 376
  8. 2016Trained Ternary Quantizationcited by 208
  9. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 145
  10. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  11. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 44
  12. 2018Training DNNs with Hybrid Block Floating Pointcited by 40
4 more
  1. 2020Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Trainingcited by 18
  2. 2020TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inferencecited by 16
  3. 2018Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimizationcited by 15
  4. 2017Low Precision RNNs: Quantizing RNNs Without Losing Accuracycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 8; 2025: 8; 2026: 4.

31 papers.

Sort:

By area:

  1. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  2. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  3. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  4. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  5. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  6. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  7. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 19
  8. 2024HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantizationcited by 17
  9. 2025MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Servingcited by 11
  10. 2022FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Supportcited by 10
  11. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
  12. 2024DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analyticscited by 8
19 more
  1. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Servingcited by 7
  2. 2023Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Designcited by 5
  3. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5
  4. 2023Number Systems for Deep Neural Network Architectures: A Surveycited by 4
  5. 2025On Stochastic Rounding with Few Random Bitscited by 4
  6. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  7. 2022Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Trainingcited by 3
  8. 2023CAMEL: Co-Designing AI Models and Embedded DRAMs for Efficient On-Device Learningcited by 3
  9. 2025Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimizationcited by 3
  10. 2025Low-Precision Streaming PCAcited by 3
  11. 2026Search Your Block Floating Point Scales!cited by 3
  12. 2023Mirage: An RNS-Based Photonic Accelerator for DNN Trainingcited by 2
  13. 2022Schr"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Trainingcited by 1
  14. 2023DAISM: Digital Approximate In-SRAM Multiplier-based Accelerator for DNN Training and Inferencecited by 1
  15. 2023BIM: Block-Wise Self-Supervised Learning with Masked Image Modelingcited by 1
  16. 2024BitQ: Tailoring Block Floating Point Precision for Improved DNN Efficiency on Resource-Constrained Devicescited by 1
  17. 2025Distribution-Aware Hadamard Quantization for Hardware-Efficient Implicit Neural Representationscited by 1
  18. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inferencecited by 1
  19. 2026MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Block Format Error Bounds and Optimal Block Size Selection
  2. 2022FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
  3. 2021Development of Quantized DNN Library for Exact Hardware Emulation
  4. 2021Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
  5. 2023Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance
  6. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
  7. 2022Adaptive Block Floating-Point for Analog Deep Learning Hardware
  8. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference