mlsanityData to 7 Oct 2026

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

arXiv 2210.17323 ↗First posted 31 Oct 2022cs.LGcited by 1,838

Not on the map. Why?

Cited by 237 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.

Key facts. Cited by 1,838 papers in the map (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

25 of 37 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
13 more
  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  3. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  6. 2021A White Paper on Neural Network Quantizationcited by 453
  7. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  8. 2019Loss Aware Post-training Quantizationcited by 94
  9. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  10. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 54
  11. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  12. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  13. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 147; 2024: 417; 2025: 542; 2026: 729.

1,838 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2024DeepSeek-V3 Technical Reportcited by 724
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  6. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  10. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  12. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  13. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  14. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  15. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  16. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  17. 2023A Survey on Model Compression for Large Language Modelscited by 263
  18. 2024Navigation World Modelscited by 262
  19. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  20. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  21. 2023Challenges and Applications of Large Language Modelscited by 251
  22. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  23. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  24. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  25. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  26. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  27. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  28. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  29. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  30. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  31. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  32. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  33. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  34. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  35. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  36. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  37. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  38. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155

50 most cited of 1,838 All 1,838 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
  2. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
  3. 2026LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization
  4. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
  5. 2026On the Expressive Power of Weight Quantization in Large Language Models
  6. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
  7. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
  8. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗