mlsanityData to 14 Oct 2026

Towards Fully FP8 GEMM LLM Training at Scale

arXiv 2505.20524 ↗First posted 26 May 2025cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at scale. Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications (GEMMs) in sensitive components, such as attention projections, compromising potential throughput gains. We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. In addition, we identify key metrics to monitor low-precision training and predict potential future divergences.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

27 of 36 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
15 more
  1. 2021Going deeper with Image Transformerscited by 764
  2. 2024Massive Activations in Large Language Modelscited by 294
  3. 2020Query-Key Normalization for Transformerscited by 280
  4. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  5. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  6. 2022FP8 Formats for Deep Learningcited by 196
  7. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  8. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  9. 2023Microscaling Data Formats for Deep Learningcited by 152
  10. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  11. 2025Transformers without Normalizationcited by 70
  12. 2024Scaling FP8 training to trillion-token LLMscited by 65
  13. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  14. 2024Deriving Activation Functions Using Integrationcited by 5
  15. 2025Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 21
  2. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 8
  3. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummiescited by 6
  4. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  5. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  6. 2025LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUscited by 2
  7. 2026Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Trainingcited by 2
  8. 2026Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4cited by 2
  9. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  10. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  11. 2026Pretraining Transformers with Quantized Softmax in Attentioncited by 0