mlsanityData to 7 Oct 2026

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats

arXiv 2307.09782 ↗First posted 19 Jul 2023cs.LG cs.AI cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

In the complex domain of large language models (LLMs), striking a balance between computational efficiency and maintaining model quality is a formidable challenge. Navigating the inherent limitations of uniform quantization, particularly when dealing with outliers, and motivated by the launch of NVIDIA's H100 hardware, this study delves into the viability of floating-point (FP) quantization, particularly focusing on FP8 and FP4, as a potential solution. Our comprehensive investigation reveals that for LLMs, FP8 activation consistently outshines its integer (INT8) equivalent, with the performance edge becoming more noticeable in models possessing parameters beyond one billion. For weight quantization, our findings indicate that FP4 exhibits comparable, if not superior, performance to INT4, simplifying deployment on FP-supported hardware like H100. To mitigate the overhead from precision alignment caused by the disparity between weights and activations, we propose two scaling constraints for weight quantization that negligibly impact the performance compared to the standard W4A8 model. We additionally enhance our quantization methods by integrating the Low Rank Compensation (LoRC) strategy, yielding improvements especially in smaller models. The results of our investigation emphasize the immense potential of FP quantization for LLMs, paving the way for high-efficiency deployment in resource-limited settings.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

27 of 38 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
15 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  2. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  3. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  4. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  5. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  6. 2022FP8 Formats for Deep Learningcited by 187
  7. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  8. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  9. 2022FP8 Quantization: The Power of the Exponentcited by 87
  10. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  11. 2016Bitwise Neural Networkscited by 29
  12. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
  13. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  14. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 18
  15. 2017Ternary Neural Networks with Fine-Grained Quantizationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 18; 2025: 9; 2026: 1.

44 papers.

Sort:

By area:

  1. 2023A Survey on Model Compression for Large Language Modelscited by 263
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  6. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  8. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 91
  9. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  10. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  12. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
32 more
  1. 2023CBQ: Cross-Block Quantization for Large Language Modelscited by 34
  2. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  3. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  4. 2023Efficient LLM Inference on CPUscited by 24
  5. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 22
  6. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  7. 2025Towards AI Search Paradigmcited by 19
  8. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  9. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  10. 2024Squat: Quant Small Language Models on the Edgecited by 16
  11. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  12. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  13. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  14. 2023A Speed Odyssey for Deployable Quantization of LLMscited by 10
  15. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Modelscited by 9
  16. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 8
  17. 2023AFPQ: Asymmetric Floating Point Quantization for LLMscited by 7
  18. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  19. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  20. 2025RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remappingcited by 6
  21. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6
  22. 2025FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformerscited by 6
  23. 2024FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigationcited by 3
  24. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  25. 2023Dual Grained Quantization: Efficient Fine-Grained Quantization for LLMcited by 2
  26. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerationscited by 2
  27. 2025ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Modelscited by 2
  28. 2025REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compressioncited by 2
  29. 2023ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformerscited by 1
  30. 2024Large Language Model Pruningcited by 1
  31. 2024Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balancecited by 0
  32. 2026MOMAT: Mixture of Multiple Atlases for Low-Power Jailbreak Defense of Quantized LLMscited by 0