# Recipes for Pre-training LLMs with MXFP8

arXiv: https://arxiv.org/abs/2506.08027 · HTML: https://mlsanity.com/paper/2506.08027/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 16 of 16)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective](https://mlsanity.com/paper/2410.05192/) (2024)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021)
- [Optimizing Large Language Model Training Using FP4 Quantization](https://mlsanity.com/paper/2501.17116/) (2025)

## Built on by (most cited first, 22 of 22)

- [Pretraining Large Language Models with NVFP4](https://mlsanity.com/paper/2509.25149/) (2025)
- [Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling](https://mlsanity.com/paper/2512.02010/) (2025)
- [INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats](https://mlsanity.com/paper/2510.25602/) (2025)
- [Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats](https://mlsanity.com/paper/2601.09555/) (2026)
- [M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization](https://mlsanity.com/paper/2601.19213/) (2026)
- [BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization](https://mlsanity.com/paper/2603.16590/) (2026)
- [SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales](https://mlsanity.com/paper/2607.20548/) (2026)
- [StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models](https://mlsanity.com/paper/2604.15416/) (2026)
- [Elucidating the Design Space of FP4 training](https://mlsanity.com/paper/2509.17791/) (2025)
- [A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models](https://mlsanity.com/paper/2509.22536/) (2025)
- [Stable FP4 Training via Transposition-Invariant Block Quantization](https://mlsanity.com/paper/2607.24953/) (2026)
- [CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding](https://mlsanity.com/paper/2511.19705/) (2025)
- [OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learning](https://mlsanity.com/paper/2605.28691/) (2026)
- [ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning](https://mlsanity.com/paper/2607.24062/) (2026)
- [Performance of Small Language Model Pretraining on FABRIC: An Empirical Study](https://mlsanity.com/paper/2602.02632/) (2026)
- [pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training](https://mlsanity.com/paper/2602.22592/) (2026)
- [MXNorm: Reusing MXFP block scales for efficient tensor normalisation](https://mlsanity.com/paper/2603.13180/) (2026)
- [Stochastic Rounding Increases Small Singular Values](https://mlsanity.com/paper/2606.00312/) (2026)
- [Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference](https://mlsanity.com/paper/2608.03867/) (2026)
- [Motif 3: Technical Report](https://mlsanity.com/paper/2608.09119/) (2026)
- [Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs](https://mlsanity.com/paper/2609.37852/) (2026)
- [Backward-State Policy Is Part of the Learning Algorithm](https://mlsanity.com/paper/2609.39813/) (2026)

## Abstract (arXiv)

Using fewer bits to represent model parameters and related tensors during pre-training has become a required technique for improving GPU efficiency without sacrificing accuracy. Microscaling (MX) formats introduced in NVIDIA Blackwell generation of GPUs represent a major advancement of this technique, making it practical to combine narrow floating-point data types with finer granularity per-block scaling factors. In turn, this enables both quantization of more tensors than previous approaches and more efficient execution of operations on those tensors. Effective use of MX-formats requires careful choices of various parameters. In this paper we review these choices and show how MXFP8-E4M3 datatype and a specific number conversion algorithm result in training sessions that match those carried out in BF16. We present results using models with up to 8B parameters, trained on high-quality datasets of up to 15T tokens.