mlsanityData to 14 Oct 2026

Quartet: Native FP4 Training Can Be Optimal for Large Language Models

arXiv 2505.14669 ↗First posted 20 May 2025cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs) models directly in low-precision offers a way to address computational costs by improving both throughput and energy efficiency. For those purposes, NVIDIA's recent Blackwell architecture facilitates very low-precision operations using FP4 variants. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on mixed-precision fallbacks. In this paper, we investigate hardware-supported FP4 training and introduce a new approach for accurate, end-to-end FP4 training with all the major computations (i.e., linear layers) in low precision. Through extensive evaluations on Llama-type models, we reveal a new low-precision scaling law that quantifies performance trade-offs across bit-widths and training setups. Guided by this investigation, we design an "optimal" technique in terms of accuracy-vs-computation, called Quartet. We implement Quartet using optimized CUDA kernels tailored for Blackwell, demonstrating that fully FP4-based training is a competitive alternative to FP16 half-precision and to FP8 training. Our code is available at https://github.com/IST-DASLab/Quartet.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

34 of 51 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2019Learned Step Size Quantizationcited by 628
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
22 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  2. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  3. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  4. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  5. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  6. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  7. 2024The rising costs of training frontier AI modelscited by 95
  8. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  9. 2024Scaling FP8 training to trillion-token LLMscited by 65
  10. 2023Training Transformers with 4-bit Integerscited by 60
  11. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  12. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 55
  13. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  14. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43
  15. 2025Training LLMs with MXFP4cited by 37
  16. 2019Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integerscited by 33
  17. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  18. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  19. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  20. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 21
  21. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 15
  22. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 33.

39 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 76
  2. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 35
  3. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  4. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 21
  5. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 18
  6. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  7. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  8. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
  9. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  10. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 5
  11. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  12. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
27 more
  1. 2025Elucidating the Design Space of FP4 trainingcited by 4
  2. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  3. 2026Adaptive Block-Scaled Data Typescited by 4
  4. 2026ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attentioncited by 4
  5. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
  6. 2026ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Trainingcited by 3
  7. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  8. 2026Normalized Architectures are Natively 4-Bitcited by 2
  9. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  10. 2026Quantizing Recursive Reasoning Modelscited by 2
  11. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 2
  12. 2026Boosting Entropy with Bell Box Quantizationcited by 1
  13. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  14. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  15. 2026Information-Theoretic Lower Bounds for Bit-Constrained Stochastic Optimization via a Reduction to Compressed Gaussian Mean Estimationcited by 1
  16. 2026M+Adam: Low-Precision Training via Additive-Multiplicative Optimizationcited by 1
  17. 2026Entropy-Constrained Adaptive Stochastic Quantizationcited by 1
  18. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0
  19. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0
  20. 2026BitNet Text Embeddingscited by 0
  21. 2026Auditing Invisible Weight Updates with Reference Tracescited by 0
  22. 2026KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantizationcited by 0
  23. 2026Quantization-Aware Pre-Training with Constrained Empirical Weight Distributioncited by 0
  24. 2026Backward-State Policy Is Part of the Learning Algorithmcited by 0
  25. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0
  26. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Trainingcited by 0
  27. 2026Mixed Precision Scaling Laws for Quantized Pretrainingcited by 0