mlsanityData to 14 Oct 2026

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

arXiv 2306.00978 ↗First posted 1 Jun 2023cs.CLcited by 1,298

Not on the map. Why?

Cited by 186 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose significant deployment challenges. We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ finds that not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To identify salient weight channels, we should refer to the activation distribution, not weights. To avoid the hardware-inefficient mix-precision quantization, we mathematically derive that scaling up the salient channels can reduce the quantization error. AWQ employs an equivalent transformation to scale the salient weight channels to protect them. The scale is determined by collecting the activation statistics offline. AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set. AWQ outperforms existing work on various language modeling and domain-specific benchmarks (coding and math). Thanks to better generalization, it achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. Alongside AWQ, we implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs. With kernel fusion and platform-aware weight packing, TinyChat offers more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs. It also democratizes the deployment of the 70B Llama-2 model on mobile GPUs.

Key facts. Cited by 1,298 papers in the map (data through 14 Oct 2026). Builds on 48 of its 69 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

48 of 69 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
36 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2019Towards VQA Models That Can Readcited by 1,718
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  13. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  14. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  15. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  17. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  18. 2023StarCoder: may the source be with you!cited by 799
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  20. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  21. 2019Learned Step Size Quantizationcited by 628
  22. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  23. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  24. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  25. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  26. 2021A White Paper on Neural Network Quantizationcited by 472
  27. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  28. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  29. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  30. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
  31. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  32. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  33. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  34. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  35. 2022TensorIR: An Abstraction for Automatic Tensorized Program Optimizationcited by 38
  36. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 43; 2024: 218; 2025: 469; 2026: 568.

1,298 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  4. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  7. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  8. 2025Training Language Models to Reason Efficientlycited by 247
  9. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  10. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  11. 2025Real-Time Execution of Action Chunking Flow Policiescited by 216
  12. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
38 more
  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2023Efficient Large Language Models: A Surveycited by 186
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  4. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  5. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  6. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  7. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  9. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  10. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  11. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  12. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  13. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  14. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  15. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  16. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  17. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  18. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  19. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  20. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  21. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  22. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  23. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  24. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  25. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  26. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  27. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  28. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  29. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  30. 2024Scaling Laws for Precisioncited by 95
  31. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  32. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  33. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  34. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  35. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 81
  36. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  37. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 79
  38. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72

50 most cited of 1,298 All 1,298 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  2. 2025On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration
  3. 2026AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
  4. 2026RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
  5. 2023A Speed Odyssey for Deployable Quantization of LLMs
  6. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
  7. 2025A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources
  8. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs