mlsanityData to 7 Oct 2026

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

arXiv 2211.10438 ↗First posted 18 Nov 2022cs.CL cs.AI cs.LGcited by 1,338

Not on the map. Why?

Cited by 188 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

Key facts. Cited by 1,338 papers in the map (data through 7 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

37 of 45 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
25 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  13. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  14. 2022Efficiently Scaling Transformer Inferencecited by 529
  15. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  16. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  18. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  19. 2021Post-Training Quantization for Vision Transformercited by 274
  20. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  21. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  22. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  23. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  24. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  25. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 130; 2024: 350; 2025: 398; 2026: 459.

1,338 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
38 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  3. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  4. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  7. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  8. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  13. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  14. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  15. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  16. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  17. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  18. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  19. 2024Massive Activations in Large Language Modelscited by 277
  20. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  21. 2023A Survey on Model Compression for Large Language Modelscited by 263
  22. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  23. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  24. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  25. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  26. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  27. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  28. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  29. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  30. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  31. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  32. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  33. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  34. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  35. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  36. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  37. 2023Efficient Large Language Models: A Surveycited by 183
  38. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175

50 most cited of 1,338 All 1,338 (JSON)