mlsanityData to 14 Oct 2026

FPTQ: Fine-grained Post-Training Quantization for Large Language Models

arXiv 2308.15987 ↗First posted 30 Aug 2023cs.CL cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

In the era of large-scale language models, the substantial parameter size poses significant challenges for deployment. Being a prevalent compression technique, quantization has emerged as the mainstream practice to tackle this issue, which is mainly centered on two recipes W8A8 and W4A16 (i.e. weights and activations in such bit widths). In this study, we propose a novel W4A8 post-training quantization method for the available open-sourced LLMs, which combines the advantages of both two recipes. Therefore, we can leverage the benefit in the I/O utilization of 4-bit weight quantization and the acceleration due to 8-bit matrix computation. Nevertheless, the W4A8 faces notorious performance degradation. As a remedy, we involve layerwise activation quantization strategies which feature a novel logarithmic equalization for most intractable layers, and we combine them with fine-grained weight quantization. Without whistles and bells, we eliminate the necessity for further fine-tuning and obtain the state-of-the-art W4A8 quantized performance on BLOOM, LLaMA, and LLaMA-2 on standard benchmarks. We confirm that the W4A8 quantization is achievable for the deployment of large language models, fostering their wide-spreading real-world applications.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

35 of 40 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2016Layer Normalizationcited by 5,669
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
23 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  14. 2021A White Paper on Neural Network Quantizationcited by 472
  15. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  16. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  17. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  18. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  19. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  20. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  21. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  22. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  23. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 6; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2025LongCat-Flash Technical Reportcited by 58
  3. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  4. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  5. 2025Towards AI Search Paradigmcited by 19
  6. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  7. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 11
  8. 2023A Speed Odyssey for Deployable Quantization of LLMscited by 10
  9. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterizationcited by 10
  10. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  11. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  12. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
4 more
  1. 2026SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipeliningcited by 4
  2. 2023Dual Grained Quantization: Efficient Fine-Grained Quantization for LLMcited by 2
  3. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Modelscited by 1
  4. 2024Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  2. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
  3. 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
  4. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
  5. 2024Post Training Quantization of Large Language Models with Microscaling Formats
  6. 2025PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models
  7. 2024CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression
  8. 2023AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models