# Training DNNs with Hybrid Block Floating Point

arXiv: https://arxiv.org/abs/1804.01526 · HTML: https://mlsanity.com/paper/1804.01526/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2018.

## Built on by (most cited first, 30 of 40)

- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [Stable and low-precision training for large-scale vision-language models](https://mlsanity.com/paper/2304.13013/) (2023)
- [Training Transformers with 4-bit Integers](https://mlsanity.com/paper/2306.11987/) (2023)
- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [LQER: Low-Rank Quantization Error Reconstruction for LLMs](https://mlsanity.com/paper/2402.02446/) (2024)
- [FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding](https://mlsanity.com/paper/2110.15456/) (2021)
- [MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization](https://mlsanity.com/paper/2411.05282/) (2024)
- [GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning](https://mlsanity.com/paper/2502.12913/) (2025)
- [Effective Interplay between Sparsity and Quantization: From Theory to Practice](https://mlsanity.com/paper/2405.20935/) (2024)
- [AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference](https://mlsanity.com/paper/2411.09909/) (2024)
- [Is Integer Arithmetic Enough for Deep Learning Training?](https://mlsanity.com/paper/2207.08822/) (2022)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving](https://mlsanity.com/paper/2510.14557/) (2025)
- [FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Support](https://mlsanity.com/paper/2203.06673/) (2022)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format](https://mlsanity.com/paper/2411.15982/) (2024)
- [DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics](https://mlsanity.com/paper/2403.14353/) (2024)
- [Dynamic Stashing Quantization for Efficient Transformer Training](https://mlsanity.com/paper/2303.05295/) (2023)
- [Multiplication-Free Transformer Training via Piecewise Affine Operations](https://mlsanity.com/paper/2305.17190/) (2023)
- [Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators](https://mlsanity.com/paper/2101.11748/) (2021)
- [Adaptive Block Floating-Point for Analog Deep Learning Hardware](https://mlsanity.com/paper/2205.06287/) (2022)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training](https://mlsanity.com/paper/2211.10737/) (2022)
- [TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture](https://mlsanity.com/paper/2411.03697/) (2024)
- [Distilling Calibration via Conformalized Credal Inference](https://mlsanity.com/paper/2501.06066/) (2025)
- [Search Your Block Floating Point Scales!](https://mlsanity.com/paper/2605.12464/) (2026)
- [Training with Mixed-Precision Floating-Point Assignments](https://mlsanity.com/paper/2301.13464/) (2023)
- [Mirage: An RNS-Based Photonic Accelerator for DNN Training](https://mlsanity.com/paper/2311.17323/) (2023)
- [EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Models](https://mlsanity.com/paper/2102.02705/) (2021)

## Abstract (arXiv)

The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5x higher throughput.