# FP8 Formats for Deep Learning

arXiv: https://arxiv.org/abs/2209.05433 · HTML: https://mlsanity.com/paper/2209.05433/

## Key facts

- Cited by 196 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [8-bit Numerical Formats for Deep Neural Networks](https://mlsanity.com/paper/2206.02915/) (2022)
- [Mixed Precision Training With 8-bit Floating Point](https://mlsanity.com/paper/1905.12334/) (2019)
- [Rethinking floating point for deep learning](https://mlsanity.com/paper/1811.01721/) (2018)
- [First-Generation Inference Accelerator Deployment at Facebook](https://mlsanity.com/paper/2107.04140/) (2021)

## Built on by (most cited first, 30 of 196)

- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://mlsanity.com/paper/2504.03624/) (2025)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)
- [AI and Memory Wall](https://mlsanity.com/paper/2403.14123/) (2024)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [FP8-LM: Training FP8 Large Language Models](https://mlsanity.com/paper/2310.18313/) (2023)
- [Stable and low-precision training for large-scale vision-language models](https://mlsanity.com/paper/2304.13013/) (2023)
- [SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training](https://mlsanity.com/paper/2505.11594/) (2025)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats](https://mlsanity.com/paper/2307.09782/) (2023)
- [MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization](https://mlsanity.com/paper/2405.17873/) (2024)
- [Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients](https://mlsanity.com/paper/2407.08296/) (2024)
- [COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training](https://mlsanity.com/paper/2410.19313/) (2024)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation](https://mlsanity.com/paper/2605.18739/) (2026)
- [Efficient Post-training Quantization with FP8 Formats](https://mlsanity.com/paper/2309.14592/) (2023)
- [u-μP: The Unit-Scaled Maximal Update Parametrization](https://mlsanity.com/paper/2407.17465/) (2024)
- [Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization](https://mlsanity.com/paper/2406.12930/) (2024)
- [Efficient LLM Inference on CPUs](https://mlsanity.com/paper/2311.00502/) (2023)
- [Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery](https://mlsanity.com/paper/2601.20088/) (2026)
- [Oscillation-Reduced MXFP4 Training for Vision Transformers](https://mlsanity.com/paper/2502.20853/) (2025)
- [Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation](https://mlsanity.com/paper/2601.22813/) (2026)
- [An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applications](https://mlsanity.com/paper/2306.02781/) (2023)

## Abstract (arXiv)

FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.