mlsanityData to 7 Oct 2026

ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks

arXiv 2312.08583 ↗First posted 14 Dec 2023cs.CL stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

This study examines 4-bit quantization methods like GPTQ in large language models (LLMs), highlighting GPTQ's overfitting and limited enhancement in Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we extend task scope to more generative categories such as code generation and abstractive summarization, in which we found that INT4 quantization can significantly underperform. However, simply shifting to higher precision formats like FP6 has been particularly challenging, thus overlooked, due to poor performance caused by the lack of sophisticated integration and system acceleration strategies on current AI hardware. Our results show that FP6, even with a coarse-grain quantization scheme, performs robustly across various algorithms and tasks, demonstrating its superiority in accuracy and versatility. Notably, with the FP6 quantization, codestar-15B model performs comparably to its FP16 counterpart in code generation, and for smaller models like the 406M it closely matches their baselines in summarization. Neither can be achieved by INT4. To better accommodate various AI hardware and achieve the best system performance, we propose a novel 4+2 design for FP6 to achieve similar latency to the state-of-the-art INT4 fine-grain quantization. With our design, FP6 can become a promising solution to the current 4-bit quantization methods used in LLMs.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 56 of its 72 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

56 of 72 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
38 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  4. 2019Learned Step Size Quantizationcited by 604
  5. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  8. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  11. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  12. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  13. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  14. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  15. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  16. 2021Post-Training Quantization for Vision Transformercited by 274
  17. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  18. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  19. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  20. 2022FP8 Formats for Deep Learningcited by 187
  21. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  22. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  23. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  24. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  25. 2022FP8 Quantization: The Power of the Exponentcited by 87
  26. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  27. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  28. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  29. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  30. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  31. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  32. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 54
  33. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  34. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  35. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  36. 2023MFTCoder: Boosting Code LLMs with Multitask Fine-Tuningcited by 34
  37. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  38. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  2. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  3. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 12
  4. 2024FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designcited by 11
  5. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  6. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 8
  7. 2024QEFT: Quantization for Efficient Fine-Tuning of LLMscited by 7
  8. 2024Scalify: scale propagation for efficient low-precision LLM trainingcited by 4
  9. 2026LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformerscited by 3
  10. 2025ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Modelscited by 2
  11. 2025FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Designcited by 2
  12. 2025Is (Selective) Round-To-Nearest Quantization All You Need?cited by 1
2 more
  1. 2025Q&C: When Quantization Meets Cache in Efficient Image Generationcited by 0
  2. 2025AMS-QUANT: Adaptive Mantissa Sharing for Floating-point Quantizationcited by 0