mlsanityData to 14 Oct 2026

Training and inference of large language models using 8-bit floating point

arXiv 2309.17224 ↗First posted 29 Sep 2023cs.LG cs.AR cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of scaling is needed to prevent degradation due to the reduced dynamic range compared to higher-precision formats. Although there exists ample literature about selecting such scalings for INT formats, this critical aspect has yet to be addressed for FP8. This paper presents a methodology to select the scalings for FP8 linear layers, based on dynamically updating per-tensor scales for the weights, gradients and activations. We apply this methodology to train and validate large language models of the type of GPT and Llama 2 using FP8, for model sizes ranging from 111M to 70B. To facilitate the understanding of the FP8 dynamics, our results are accompanied by plots of the per-tensor scale distribution for weights, activations and gradients during both training and inference.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

24 of 37 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2017Mixed Precision Trainingcited by 1,156
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2022Efficiently Scaling Transformer Inferencecited by 539
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
12 more
  1. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  2. 2022FP8 Formats for Deep Learningcited by 196
  3. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  4. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  5. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  6. 2022FP8 Quantization: The Power of the Exponentcited by 88
  7. 2023Intriguing Properties of Quantization at Scalecited by 49
  8. 2019Dissecting the Graphcore IPU Architecture via Microbenchmarkingcited by 47
  9. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  10. 20228-bit Numerical Formats for Deep Neural Networkscited by 25
  11. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  12. 2018Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seqcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 5; 2026: 5.

20 papers.

Sort:

By area:

  1. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  2. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  3. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  4. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  5. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  6. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  7. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 10
  8. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  9. 2024S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-trainingcited by 7
  10. 2024Ascend HiFloat8 Format for Deep Learningcited by 7
  11. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  12. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
8 more
  1. 2023Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAscited by 4
  2. 2024InfoTech Assistant: A Multimodal Conversational Agent for InfoTechnology Web Portal Queriescited by 1
  3. 2025pychop: Emulating Low-Precision Arithmetic in Numerical Methods and Neural Networkscited by 1
  4. 2024OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruningcited by 0
  5. 2026On the Spectral Flattening of Quantized Embeddingscited by 0
  6. 2026Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignmentcited by 0
  7. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0
  8. 2026GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Trainingcited by 0