mlsanityData to 13 Oct 2026

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

arXiv 2609.02846 ↗First posted 2 Sep 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine nv recipe addresses this with current-tensor scaling, a randomized Hadamard transform (RHT), and bfloat16 (BF16) final layers, adding work outside the FP4 matrix multiplications. We instead pair E2M1 payloads with unsigned E5M3 (ue) block scales. Their wider range permits periodic tensor scaling, while our recipe applies selective stochastic rounding to backward gradients, omits RHT, and uses FP4 in all eligible internal linears. We pretrain a Nemotron-H 8B model for nearly 190 billion tokens. Compared with Transformer Engine nv, the proposed block-16 recipe finishes with lower final-window training loss and, under their respective quantized-inference policies, lower validation loss measured as held-out negative log-likelihood. Its quantized-inference downstream point estimates are also higher on all three reported aggregates. A native nv execution ablation that jointly removes RHT and the BF16 final-block exemption increases measured model-body token throughput by 21.2%. These results demonstrate end-to-end software-emulated uefp pretraining with a simpler recipe and motivate native support for ue block scaling.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

29 of 38 references.

References · 29

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,799
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 614
  4. 2015Deep Learning with Limited Numerical Precisioncited by 372
  5. 2022FP8 Formats for Deep Learningcited by 192
  6. 2023Microscaling Data Formats for Deep Learningcited by 146
  7. 2024OLMES: A Standard for Language Model Evaluationscited by 101
  8. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 93
  9. 2025Pretraining Large Language Models with NVFP4cited by 71
  10. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 55
  11. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  12. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 38
17 more
  1. 2025Training LLMs with MXFP4cited by 36
  2. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 33
  3. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  4. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 21
  5. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 20
  6. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  7. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  8. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  9. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  10. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Modelscited by 8
  11. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 8
  12. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  13. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  14. 2026Pretraining large language models with MXFP4 on Native FP4 Hardwarecited by 5
  15. 2025On Stochastic Rounding with Few Random Bitscited by 4
  16. 2025Elucidating the Design Space of FP4 trainingcited by 4
  17. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Hardware-Aware FP4 FlashAttention-4cited by 2
  2. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  2. 2025FP4 All the Way: Fully Quantized Training of LLMs
  3. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
  4. 2026Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
  5. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formats
  6. 2025Block Rotation is All You Need for MXFP4 Quantization
  7. 2026MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations
  8. 2026Normalized Architectures are Natively 4-Bit