mlsanityData to 7 Oct 2026

FP8-LM: Training FP8 Large Language Models

arXiv 2310.18313 ↗First posted 27 Oct 2023cs.LG cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore FP8 low-bit data formats for efficient training of large language models (LLMs). Our key insight is that most variables, such as gradients and optimizer states, in LLM training can employ low-precision data formats without compromising model accuracy and requiring no changes to hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision framework for training LLMs. This framework offers three levels of FP8 utilization to streamline mixed-precision and distributed parallel training for LLMs. It gradually incorporates 8-bit gradients, optimizer states, and distributed learning in an incremental manner. Experiment results show that, during the training of GPT-175B model on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a remarkable 39% reduction in real memory usage but also ran 75% faster than the widely adopted BF16 framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer Engine by 37%. This largely reduces the training costs for large foundation models. Furthermore, our FP8 mixed-precision training methodology is generic. It can be seamlessly applied to other tasks such as LLM instruction tuning and reinforcement learning with human feedback, offering savings in fine-tuning expenses. Our FP8 low-precision training framework is open-sourced at {https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 52 of its 74 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

52 of 74 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  13. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  14. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  15. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  16. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  18. 2023PaLM 2 Technical Reportcited by 1,154
  19. 2017Mixed Precision Trainingcited by 1,135
  20. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  21. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  22. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  23. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  25. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  26. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  27. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  28. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  29. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  30. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  31. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  32. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  33. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  34. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  35. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  36. 2018A Simple Method for Commonsense Reasoningcited by 223
  37. 2022FP8 Formats for Deep Learningcited by 187
  38. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 30; 2026: 25.

75 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 724
  2. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  3. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  4. 2024Scaling Laws for Precisioncited by 92
  5. 2024Scaling FP8 training to trillion-token LLMscited by 63
  6. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  7. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  8. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
  9. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  10. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  11. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  12. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 28
38 more
  1. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  2. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  3. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 20
  4. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 16
  5. 2024ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Lawscited by 15
  6. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  7. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
  8. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  9. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 13
  10. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
  11. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  12. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  13. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Modelscited by 10
  14. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 10
  15. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  16. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  17. 2024Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planningcited by 7
  18. 2024LoCo: Low-Bit Communication Adaptor for Large-scale Model Trainingcited by 5
  19. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 5
  20. 2024Accurate Block Quantization in LLMs with Outlierscited by 4
  21. 2024Scalify: scale propagation for efficient low-precision LLM trainingcited by 4
  22. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  23. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  24. 2025Compute-Optimal Quantization-Aware Trainingcited by 4
  25. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  26. 2026FlashOptim: Optimizers for Memory-Efficient Trainingcited by 4
  27. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  28. 2024EXAQ: Exponent Aware Quantization For LLMs Accelerationcited by 3
  29. 2025A Review of DeepSeek Models' Key Innovative Techniquescited by 3
  30. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Modelscited by 3
  31. 2025A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Modelscited by 3
  32. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 3
  33. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  34. 2024Towards Safe Multilingual Frontier AIcited by 2
  35. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  36. 2025MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scalingcited by 2
  37. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  38. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient Post-training Quantization with FP8 Formats
  2. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
  3. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  4. 2026L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
  5. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
  6. 2026BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
  7. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospects
  8. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism