mlsanityData to 7 Oct 2026

FP8 Formats for Deep Learning

arXiv 2209.05433 ↗First posted 12 Sep 2022cs.LGcited by 187

Not on the map. Why?

Abstract (arXiv)

FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

20 of 25 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  12. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
8 more
  1. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  2. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  3. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 97
  4. 2022FP8 Quantization: The Power of the Exponentcited by 87
  5. 20228-bit Numerical Formats for Deep Neural Networkscited by 23
  6. 2018Rethinking floating point for deep learningcited by 22
  7. 2019Mixed Precision Training With 8-bit Floating Pointcited by 22
  8. 2021First-Generation Inference Accelerator Deployment at Facebookcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 24; 2024: 30; 2025: 49; 2026: 82.

187 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 724
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  4. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  5. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2024Scaling Laws for Precisioncited by 92
  8. 2024AI and Memory Wallcited by 89
  9. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  10. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  11. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  12. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
38 more
  1. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  2. 2024Scaling FP8 training to trillion-token LLMscited by 63
  3. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  4. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  5. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  6. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantizationcited by 42
  7. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  8. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  9. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  10. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  11. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  12. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 25
  13. 2023Efficient LLM Inference on CPUscited by 24
  14. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 22
  15. 2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applicationscited by 20
  16. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  17. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  18. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 20
  19. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  20. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 18
  21. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  22. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  23. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  24. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 16
  25. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 16
  26. 2023Revisiting Neural Retrieval on Acceleratorscited by 14
  27. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  28. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  29. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giantcited by 12
  30. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  31. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  32. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  33. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  34. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resourcescited by 10
  35. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 10
  36. 2023Guaranteed Approximation Bounds for Mixed-Precision Neural Operatorscited by 9
  37. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  38. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniquescited by 9

50 most cited of 187 All 187 (JSON)