# 8-bit Numerical Formats for Deep Neural Networks

arXiv: https://arxiv.org/abs/2206.02915 · HTML: https://mlsanity.com/paper/2206.02915/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Mixed Precision Training of Convolutional Neural Networks using Integer Operations](https://mlsanity.com/paper/1802.00930/) (2018)
- [Mixed Precision Training With 8-bit Floating Point](https://mlsanity.com/paper/1905.12334/) (2019)
- [Rethinking floating point for deep learning](https://mlsanity.com/paper/1811.01721/) (2018)
- [Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq](https://mlsanity.com/paper/1805.10387/) (2018)
- [Adaptive Loss Scaling for Mixed Precision Training](https://mlsanity.com/paper/1910.12385/) (2019)
- [Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training](https://mlsanity.com/paper/2106.03640/) (2021)

## Built on by (most cited first, 25 of 25)

- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [Efficient Post-training Quantization with FP8 Formats](https://mlsanity.com/paper/2309.14592/) (2023)
- [u-μP: The Unit-Scaled Maximal Update Parametrization](https://mlsanity.com/paper/2407.17465/) (2024)
- [Training and inference of large language models using 8-bit floating point](https://mlsanity.com/paper/2309.17224/) (2023)
- [Unit Scaling: Out-of-the-Box Low-Precision Training](https://mlsanity.com/paper/2303.11257/) (2023)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [A Performance Evaluation of a Quantized Large Language Model on Various Smartphones](https://mlsanity.com/paper/2312.12472/) (2023)
- [StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models](https://mlsanity.com/paper/2604.15416/) (2026)
- [RedMule: A Mixed-Precision Matrix-Matrix Operation Engine for Flexible and Energy-Efficient On-Chip Linear Algebra and TinyML Training Acceleration](https://mlsanity.com/paper/2301.03904/) (2023)
- [Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs](https://mlsanity.com/paper/2311.12359/) (2023)
- [Low-Precision Mixed-Computation Models for Inference on Edge](https://mlsanity.com/paper/2312.02210/) (2023)
- [Scalify: scale propagation for efficient low-precision LLM training](https://mlsanity.com/paper/2407.17353/) (2024)
- [On Stochastic Rounding with Few Random Bits](https://mlsanity.com/paper/2504.20634/) (2025)
- [DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models](https://mlsanity.com/paper/2507.09955/) (2025)
- [Compute Requirements for Algorithmic Innovation in Frontier AI Models](https://mlsanity.com/paper/2507.10618/) (2025)
- [Just-in-time Quantization with Processing-In-Memory for Efficient ML Training](https://mlsanity.com/paper/2311.05034/) (2023)
- [Optimal Formats for Weight Quantisation](https://mlsanity.com/paper/2505.12988/) (2025)
- [Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers](https://mlsanity.com/paper/2602.02707/) (2026)
- [Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learning](https://mlsanity.com/paper/2606.04028/) (2026)
- [Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integration](https://mlsanity.com/paper/2511.06313/) (2025)
- [Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats](https://mlsanity.com/paper/2606.09686/) (2026)
- [Hadamard-Domain Model Quantization for Learned Image Coding](https://mlsanity.com/paper/2608.01653/) (2026)

## Abstract (arXiv)

Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.