mlsanityData to 14 Oct 2026

Training Deep Neural Networks with 8-bit Floating Point Numbers

arXiv 1812.08011 ↗First posted 19 Dec 2018cs.LG stat.MLcited by 164

Not on the map. Why?

Abstract (arXiv)

The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.

Key facts. Cited by 164 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 35; 2022: 24; 2023: 33; 2024: 39; 2025: 24; 2026: 9.

164 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  4. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  5. 2021NeRV: Neural Representations for Videoscited by 256
  6. 2022FP8 Formats for Deep Learningcited by 196
  7. 2022FP8 Quantization: The Power of the Exponentcited by 88
  8. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  9. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  10. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  11. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  12. 2023Training Transformers with 4-bit Integerscited by 60
38 more
  1. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  2. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  3. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  4. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 41
  5. 2024Tiny Machine Learning: Progress and Futurescited by 40
  6. 2022Hardware Approximate Techniques for Deep Neural Network Accelerators: A Surveycited by 39
  7. 2023A Survey on Efficient Training of Transformerscited by 39
  8. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  10. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  11. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  12. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  13. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  14. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  15. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  16. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  17. 2021Distribution Adaptive INT8 Quantization for Training CNNscited by 28
  18. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  19. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  20. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  21. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  22. 20228-bit Numerical Formats for Deep Neural Networkscited by 25
  23. 2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
  24. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  25. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  26. 2021CPT: Efficient Deep Neural Network Training via Cyclic Precisioncited by 19
  27. 2021Charged particle tracking via edge-classifying interaction networkscited by 19
  28. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  29. 2021LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networkscited by 19
  30. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 19
  31. 2021Distributed Learning Systems with First-order Methodscited by 17
  32. 2021Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applicationscited by 17
  33. 2023Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learningcited by 17
  34. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  35. 2021Secure Quantized Training for Deep Learningcited by 15
  36. 2022Low-Precision Stochastic Gradient Langevin Dynamicscited by 12
  37. 2023A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidancecited by 12
  38. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12

50 most cited of 164 All 164 (JSON)