mlsanityData to 14 Oct 2026

FP4 All the Way: Fully Quantized Training of LLMs

arXiv 2505.19115 ↗First posted 25 May 2025cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quantized training: when the gradient norm falls below approximately √3 times the quantization noise, quantized training becomes less effective. Leveraging these insights, we successfully train a 7-billion-parameter model on 256 Intel Gaudi2 accelerators. The resulting FP4-trained model achieves downstream task performance comparable to a standard BF16 baseline, confirming that FP4 training is a practical and highly efficient approach for large-scale LLM training. A reference implementation is supplied in https://github.com/Anonymous1252022/fp4-all-the-way .

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

19 of 25 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  10. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  11. 2023Microscaling Data Formats for Deep Learningcited by 152
  12. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
7 more
  1. 2024Scaling FP8 training to trillion-token LLMscited by 65
  2. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  3. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  4. 2025Training LLMs with MXFP4cited by 37
  5. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  6. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  7. 2019Towards Automatic Concept-based Explanationscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 32.

40 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 76
  2. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 35
  3. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  4. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  5. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 21
  6. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  7. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 15
  8. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 8
  9. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
  10. 2026High-Rate Quantized Matrix Multiplication Icited by 7
  11. 2025TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edgecited by 6
  12. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
28 more
  1. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 5
  2. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  3. 2026Pretraining large language models with MXFP4 on Native FP4 Hardwarecited by 5
  4. 2025Elucidating the Design Space of FP4 trainingcited by 4
  5. 2026Adaptive Block-Scaled Data Typescited by 4
  6. 2026ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attentioncited by 4
  7. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
  8. 2026Search Your Block Floating Point Scales!cited by 3
  9. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  10. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  11. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Trainingcited by 2
  12. 2026Practical FP4 Training for Large-Scale MoE Models on Hopper GPUscited by 2
  13. 2026Normalized Architectures are Natively 4-Bitcited by 2
  14. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  15. 2026Quantizing Recursive Reasoning Modelscited by 2
  16. 2026The Curse and Blessing of Mean Bias in FP4-Quantized LLM Trainingcited by 1
  17. 2026LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIMcited by 1
  18. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  19. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  20. 2025Convergence for Discrete Parameter Update Schemescited by 0
  21. 2025MoR: Mixture Of Representations For Mixed-Precision Trainingcited by 0
  22. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0
  23. 2026Not All NVFP4 QAT Recipes Are Equal: How Architecture and Scale Shape Model Quality for Anomaly Segmentationcited by 0
  24. 2026GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inferencecited by 0
  25. 2026KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantizationcited by 0
  26. 2026MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inferencecited by 0
  27. 2026Backward-State Policy Is Part of the Learning Algorithmcited by 0
  28. 2026Mixed Precision Scaling Laws for Quantized Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
  2. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  3. 2026FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
  4. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformers
  5. 2026Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4
  6. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
  7. 2023Understanding the Impact of Post-Training Quantization on Large Language Models
  8. 2026OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization