mlsanityData to 7 Oct 2026

Training LLMs with MXFP4

arXiv 2502.20586 ↗First posted 27 Feb 2025cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training significantly degrades model quality. In this work, we present the first near-lossless training recipe that uses MXFP4 GEMMs, which are 2× faster than FP8 on supported hardware. Our key insight is to compute unbiased gradient estimates with stochastic rounding (SR), resulting in more accurate model updates. However, directly applying SR to MXFP4 can result in high variance from block-level outliers, harming convergence. To overcome this, we use the random Hadamard tranform to theoretically bound the variance of SR. We train GPT models up to 6.7B parameters and find that our method induces minimal degradation over mixed-precision BF16 training. Our recipe computes >1/2 the training FLOPs in MXFP4, enabling an estimated speedup of >1.3× over FP8 and >1.7× over BF16 during backpropagation.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 5 of its 15 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

5 of 15 references.

References · 5

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2023Training Transformers with 4-bit Integerscited by 58
  5. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 23.

36 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 67
  2. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  3. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
  4. 2025Model-Preserving Adaptive Roundingcited by 20
  5. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  6. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  7. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 13
  8. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
  9. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  10. 2026DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantizationcited by 8
  11. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  12. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
24 more
  1. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 4
  2. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 3
  3. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  4. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  5. 2025OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Accelerationcited by 2
  6. 2026L3: Large Lookup Layerscited by 2
  7. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2
  8. 2026Practical FP4 Training for Large-Scale MoE Models on Hopper GPUscited by 2
  9. 2026Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4cited by 2
  10. 2026VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computationcited by 2
  11. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  12. 2026ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Trainingcited by 2
  13. 2026The Curse and Blessing of Mean Bias in FP4-Quantized LLM Trainingcited by 1
  14. 2026Information-Theoretic Lower Bounds for Bit-Constrained Stochastic Optimization via a Reduction to Compressed Gaussian Mean Estimationcited by 1
  15. 2026CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limitscited by 1
  16. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 1
  17. 2025Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integrationcited by 0
  18. 2026Performance of Small Language Model Pretraining on FABRIC: An Empirical Studycited by 0
  19. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0
  20. 2026MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocationcited by 0
  21. 2026Not All NVFP4 QAT Recipes Are Equal: How Architecture and Scale Shape Model Quality for Anomaly Segmentationcited by 0
  22. 2026Stochastic Rounding Increases Small Singular Valuescited by 0
  23. 2026Auditing Invisible Weight Updates with Reference Tracescited by 0
  24. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0