mlsanityData to 14 Oct 2026

FP8 Quantization: The Power of the Exponent

arXiv 2208.09225 ↗First posted 19 Aug 2022cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have an extra degree of freedom, assigning some bits to work on an exponential scale instead. This paper in-depth investigates this benefit of the floating point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance. Then we show how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and the number of exponent bits in the FP8 format. Our chief conclusion is that when doing post-training quantization for a wide range of networks, the FP8 format is better than INT8 in terms of accuracy, and the choice of the number of exponent bits is driven by the severity of outliers in the network. We also conduct experiments with quantization-aware training where the difference in formats disappears as the network is trained to reduce the effect of outliers.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 19 Aug 2022.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
19 more
  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  2. 2019Learned Step Size Quantizationcited by 628
  3. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  4. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  5. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  6. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  7. 2021A White Paper on Neural Network Quantizationcited by 472
  8. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  9. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  10. 2015Deep Learning with Limited Numerical Precisioncited by 376
  11. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  12. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  13. 2020SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Drivingcited by 226
  14. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  15. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  16. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  17. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 143
  18. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  19. 2021All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 21; 2024: 23; 2025: 20; 2026: 22.

88 papers.

Sort:

By area:

  1. 2022FP8 Formats for Deep Learningcited by 196
  2. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  3. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  4. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  5. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  8. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  9. 2023Intriguing Properties of Quantization at Scalecited by 49
  10. 2023Pruning vs Quantization: Which is Better?cited by 46
  11. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  12. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
38 more
  1. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 32
  2. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  3. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  4. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  5. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 28
  6. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  7. 2023Efficient LLM Inference on CPUscited by 24
  8. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  9. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 19
  10. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 19
  11. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  12. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  13. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  14. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  15. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  16. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  17. 2024BASS: Batched Attention-optimized Speculative Samplingcited by 11
  18. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 11
  19. 2024GPTCast: a weather language model for precipitation nowcastingcited by 8
  20. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  21. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Modelscited by 8
  22. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 8
  23. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMscited by 7
  24. 2023FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Searchcited by 6
  25. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  26. 2024TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Modelscited by 6
  27. 2025FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformerscited by 6
  28. 2025Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuningcited by 6
  29. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Modelscited by 6
  30. 2025Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scalecited by 5
  31. 2023Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAscited by 4
  32. 2023FP8-BERT: Post-Training Quantization for Transformercited by 4
  33. 2024Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulatorscited by 4
  34. 2024Scalify: scale propagation for efficient low-precision LLM trainingcited by 4
  35. 2026SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipeliningcited by 4
  36. 2023Low-Precision Floating-Point for Efficient On-Board Deep Neural Network Processingcited by 3
  37. 2024Towards Narrowing the Generalization Gap in Deep Boolean Networkscited by 3
  38. 2025GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generationcited by 3

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization
  2. 2025Low-bit Model Quantization for Deep Neural Networks: A Survey
  3. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
  4. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
  5. 2024FP=xINT:Representing Neural Networks via Low-Bit Series Basis Functions
  6. 2021Neural Network Quantization for Efficient Inference: A Survey
  7. 2023NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search
  8. 2021Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms