mlsanityData to 14 Oct 2026

Scaling FP8 training to trillion-token LLMs

arXiv 2409.12517 ↗First posted 19 Sep 2024cs.LG cs.AIcited by 65

Not on the map. Why?

Abstract (arXiv)

We train, for the first time, large language models using FP8 precision on datasets up to 2 trillion tokens -- a 20-fold increase over previous limits. Through these extended training runs, we uncover critical instabilities in FP8 training that were not observable in earlier works with shorter durations. We trace these instabilities to outlier amplification by the SwiGLU activation function. Interestingly, we show, both analytically and empirically, that this amplification happens only over prolonged training periods, and link it to a SwiGLU weight alignment process. To address this newly identified issue, we introduce Smooth-SwiGLU, a novel modification that ensures stable FP8 training without altering function behavior. We also demonstrate, for the first time, FP8 quantization of both Adam optimizer moments. Combining these innovations, we successfully train a 7B parameter model using FP8 precision on 256 Intel Gaudi2 accelerators, achieving on-par results with the BF16 baseline while delivering up to a ~ 34 % throughput improvement. A reference implementation is supplied in https://github.com/Anonymous1252022/Megatron-DeepSpeed.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

11 of 14 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  9. 2022FP8 Formats for Deep Learningcited by 196
  10. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  11. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 30; 2026: 30.

65 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 811
  2. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  3. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  4. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 40
  5. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  6. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 34
  7. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  8. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 27
  9. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  10. 2025Recipes for Pre-training LLMs with MXFP8cited by 22
  11. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
  12. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 19
38 more
  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  2. 2025Scaling Law for Quantization-Aware Trainingcited by 17
  3. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 15
  4. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  5. 2025FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networkscited by 12
  6. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  7. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  8. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  9. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 8
  10. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
  11. 2026Generative Recommendation for Large-Scale Advertisingcited by 7
  12. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummiescited by 6
  13. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  14. 2024Deriving Activation Functions Using Integrationcited by 5
  15. 2025μnit Scaling: Simple and Scalable FP8 LLM Trainingcited by 5
  16. 2025Stability of Transformers under Layer Normalizationcited by 5
  17. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 5
  18. 2026FlashOptim: Optimizers for Memory-Efficient Trainingcited by 5
  19. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  20. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
  21. 2025Elucidating the Design Space of FP4 trainingcited by 4
  22. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  23. 2025A Review of DeepSeek Models' Key Innovative Techniquescited by 3
  24. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Modelscited by 3
  25. 2025A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Modelscited by 3
  26. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
  27. 2025Faster Inference of LLMs using FP8 on the Intel Gaudicited by 2
  28. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Trainingcited by 2
  29. 2025MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scalingcited by 2
  30. 2025LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUscited by 2
  31. 2026An Empirical Study on Noisy Data and LLM Pretraining Loss Divergencecited by 2
  32. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 2
  33. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  34. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  35. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  36. 2026Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendationcited by 1
  37. 2026M+Adam: Low-Precision Training via Additive-Multiplicative Optimizationcited by 1
  38. 2025Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Trainingcited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  2. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
  3. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
  4. 2025FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error
  5. 2023Efficient Post-training Quantization with FP8 Formats
  6. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning
  7. 2026Pretraining large language models with MXFP4 on Native FP4 Hardware
  8. 2025FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration