mlsanityData to 7 Oct 2026

Recipes for Pre-training LLMs with MXFP8

arXiv 2506.08027 ↗First posted 30 May 2025cs.LG cs.AI cs.DCcited by 18

Not on the map. Why?

Abstract (arXiv)

Using fewer bits to represent model parameters and related tensors during pre-training has become a required technique for improving GPU efficiency without sacrificing accuracy. Microscaling (MX) formats introduced in NVIDIA Blackwell generation of GPUs represent a major advancement of this technique, making it practical to combine narrow floating-point data types with finer granularity per-block scaling factors. In turn, this enables both quantization of more tensors than previous approaches and more efficient execution of operations on those tensors. Effective use of MX-formats requires careful choices of various parameters. In this paper we review these choices and show how MXFP8-E4M3 datatype and a specific number conversion algorithm result in training sessions that match those carried out in BF16. We present results using models with up to 8B parameters, trained on high-quality datasets of up to 15T tokens.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

16 of 26 references.

References · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  12. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
4 more
  1. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  2. 2024Scaling FP8 training to trillion-token LLMscited by 63
  3. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  4. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 12.

18 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 67
  2. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
  3. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  4. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 8
  5. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 5
  6. 2025Elucidating the Design Space of FP4 trainingcited by 4
  7. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  8. 2025A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Modelscited by 3
  9. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  10. 2025CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Roundingcited by 1
  11. 2026OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learningcited by 1
  12. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
6 more
  1. 2026Performance of Small Language Model Pretraining on FABRIC: An Empirical Studycited by 0
  2. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Trainingcited by 0
  3. 2026MXNorm: Reusing MXFP block scales for efficient tensor normalisationcited by 0
  4. 2026Stochastic Rounding Increases Small Singular Valuescited by 0
  5. 2026ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learningcited by 0
  6. 2026Backward-State Policy Is Part of the Learning Algorithmcited by 0