mlsanityData to 7 Oct 2026

Format-Aware Fusion for Fast FP4 Pretraining

arXiv 2610.00053 ↗First posted 4 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gain. We present format-aware fusion, which co-designs each quantization producer with its scale domain and consumer layout for native mxfp, global nvfp, and cooperative-thread-array-local nvfp. We evaluate Llama-3-family 8B pretraining through 160 billion tokens using bfloat16 output projections and compiled cross entropy. In matched same-accelerator probes, bfloat16 and Transformer Engine nvfp reach 18.8K and 27.6K tokens/s/GPU, while our fastest custom route reaches 37.9K. mxfp with row-gradient stochastic rounding and fixed-sign 32-value Hadamard weight-gradient preconditioning reaches 37.2K tokens/s/GPU (86.3% bfloat16 model FLOP utilization) and ends 2.11% above the raw bfloat16 training-loss endpoint. A Transformer Engine recipe with four final bfloat16 blocks ends 0.87% above bfloat16 at 27.1K tokens/s/GPU. Downstream rankings differ from training-loss rankings, showing that FP4 outcomes depend jointly on scale contract, operand, and execution path.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 48 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

41 of 48 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  7. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  8. 2015Deep Learning with Limited Numerical Precisioncited by 368
  9. 20242 OLMo 2 Furiouscited by 302
  10. 2022FP8 Formats for Deep Learningcited by 187
  11. 2023Microscaling Data Formats for Deep Learningcited by 132
  12. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
29 more
  1. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  2. 2025Pretraining Large Language Models with NVFP4cited by 67
  3. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  4. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  5. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  6. 2025Training LLMs with MXFP4cited by 36
  7. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35
  8. 2024ThunderKittens: Simple, Fast, and Adorable AI Kernelscited by 34
  9. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
  10. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  11. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  12. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 16
  13. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
  14. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  15. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  16. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  17. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Modelscited by 8
  18. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 7
  19. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  20. 2026Pretraining large language models with MXFP4 on Native FP4 Hardwarecited by 5
  21. 2026CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programscited by 5
  22. 2025On Stochastic Rounding with Few Random Bitscited by 4
  23. 2025Elucidating the Design Space of FP4 trainingcited by 4
  24. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 4
  25. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  26. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2
  27. 2026Practical FP4 Training for Large-Scale MoE Models on Hopper GPUscited by 2
  28. 2026Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipecited by 1
  29. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 1

Built on by

Not cited yet.