mlsanityData to 14 Oct 2026

8-bit Numerical Formats for Deep Neural Networks

arXiv 2206.02915 ↗First posted 6 Jun 2022cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

29 of 49 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2017Decoupled Weight Decay Regularizationcited by 1,831
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
17 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  4. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  5. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  7. 2015Deep Learning with Limited Numerical Precisioncited by 376
  8. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  9. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  10. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  11. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 99
  12. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  13. 2019Mixed Precision Training With 8-bit Floating Pointcited by 23
  14. 2018Rethinking floating point for deep learningcited by 22
  15. 2018Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seqcited by 11
  16. 2019Adaptive Loss Scaling for Mixed Precision Trainingcited by 7
  17. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 3; 2025: 6; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 811
  2. 2022FP8 Formats for Deep Learningcited by 196
  3. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  4. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  5. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  6. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  7. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  8. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  9. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  10. 2023A Performance Evaluation of a Quantized Large Language Model on Various Smartphonescited by 7
  11. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  12. 2023RedMule: A Mixed-Precision Matrix-Matrix Operation Engine for Flexible and Energy-Efficient On-Chip Linear Algebra and TinyML Training Accelerationcited by 4
13 more
  1. 2023Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAscited by 4
  2. 2023Low-Precision Mixed-Computation Models for Inference on Edgecited by 4
  3. 2024Scalify: scale propagation for efficient low-precision LLM trainingcited by 4
  4. 2025On Stochastic Rounding with Few Random Bitscited by 4
  5. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Modelscited by 3
  6. 2025Compute Requirements for Algorithmic Innovation in Frontier AI Modelscited by 2
  7. 2023Just-in-time Quantization with Processing-In-Memory for Efficient ML Trainingcited by 1
  8. 2025Optimal Formats for Weight Quantisationcited by 1
  9. 2026Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformerscited by 1
  10. 2026Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learningcited by 1
  11. 2025Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integrationcited by 0
  12. 2026Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formatscited by 0
  13. 2026Hadamard-Domain Model Quantization for Learned Image Codingcited by 0