mlsanityData to 14 Oct 2026

Optimizing Large Language Model Training Using FP4 Quantization

arXiv 2501.17116 ↗First posted 28 Jan 2025cs.LG cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 51 of its 63 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

51 of 63 references.

References · 51

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
38 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  12. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  13. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  14. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  15. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  16. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  17. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  18. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  19. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  20. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  21. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  22. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  23. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  24. 2023Microscaling Data Formats for Deep Learningcited by 152
  25. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  26. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  27. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  28. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  29. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  30. 2024Scaling FP8 training to trillion-token LLMscited by 65
  31. 2023Training Transformers with 4-bit Integerscited by 60
  32. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  33. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37
  34. 2019Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integerscited by 33
  35. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  36. 2022SDQ: Stochastic Differentiable Quantization with Mixed Precisioncited by 28
  37. 2019Mixed Precision Training With 8-bit Floating Pointcited by 23
  38. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 29.

56 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 76
  2. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 40
  3. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  4. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 35
  5. 2025Recipes for Pre-training LLMs with MXFP8cited by 22
  6. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
  7. 2025Scaling Law for Quantization-Aware Trainingcited by 17
  8. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  9. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  10. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  11. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 8
  12. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
38 more
  1. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  2. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  3. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  4. 2026Pretraining large language models with MXFP4 on Native FP4 Hardwarecited by 5
  5. 2025Elucidating the Design Space of FP4 trainingcited by 4
  6. 2025Compute-Optimal Quantization-Aware Trainingcited by 4
  7. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  8. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
  9. 2026Search Your Block Floating Point Scales!cited by 3
  10. 2026LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantizationcited by 3
  11. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  12. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  13. 2025FraQAT: Quantization Aware Training with Fractional bitscited by 2
  14. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Trainingcited by 2
  15. 2026Normalized Architectures are Natively 4-Bitcited by 2
  16. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 2
  17. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 2
  18. 2025FPQVAR: Floating Point Quantization for Visual Autoregressive Model with FPGA Hardware Co-designcited by 1
  19. 2025MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Modelscited by 1
  20. 2025FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantizationcited by 1
  21. 2025Learning the Topic, Not the Language: How LLMs Classify Online Immigration Discourse Across Languagescited by 1
  22. 2025Approximate Bayesian Inference via Bitstring Representationscited by 1
  23. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  24. 2026The Curse and Blessing of Mean Bias in FP4-Quantized LLM Trainingcited by 1
  25. 2026Fitting Is Not Enough: Smoothness in Extremely Quantized LLMscited by 1
  26. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  27. 2026Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMscited by 1
  28. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  29. 2025Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Trainingcited by 0
  30. 2025Variational Inference for Quantum HyperNetworkscited by 0
  31. 2025AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuningcited by 0
  32. 2025SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributionscited by 0
  33. 2025A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agentscited by 0
  34. 2026HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMscited by 0
  35. 2026Performance of Small Language Model Pretraining on FABRIC: An Empirical Studycited by 0
  36. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0
  37. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0
  38. 2026Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
  2. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  3. 2023Understanding the Impact of Post-Training Quantization on Large Language Models
  4. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  5. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  6. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
  7. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Models
  8. 2026FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling