mlsanityData to 14 Oct 2026

SqueezeLLM: Dense-and-Sparse Quantization

arXiv 2306.07629 ↗First posted 13 Jun 2023cs.CL cs.LGcited by 294

Not on the map. Why?

Cited by 47 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challenge due to their unprecedented resource requirements. This has forced existing deployment frameworks to use multi-GPU inference pipelines, which are often complex and costly, or to use smaller and less performant models. In this work, we demonstrate that the main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, specifically for single batch inference. While quantization has emerged as a promising solution by representing weights with reduced precision, previous efforts have often resulted in notable performance degradation. To address this, we introduce SqueezeLLM, a post-training quantization framework that not only enables lossless compression to ultra-low precisions of up to 3-bit, but also achieves higher quantization performance under the same memory constraint. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format. When applied to the LLaMA models, our 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory requirement. Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup compared to the baseline. Our code is available at https://github.com/SqueezeAILab/SqueezeLLM.

Key facts. Cited by 294 papers in the map (data through 14 Oct 2026). Builds on 75 of its 103 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

75 of 103 references.

References · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2020Reformer: The Efficient Transformercited by 1,668
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  12. 2019Are Sixteen Heads Really Better than One?cited by 816
  13. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  14. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  15. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  16. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  17. 2019The State of Sparsity in Deep Neural Networkscited by 476
  18. 2017Non-Autoregressive Neural Machine Translationcited by 463
  19. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  20. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  21. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  22. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  23. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  24. 2022Confident Adaptive Language Modelingcited by 310
  25. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  26. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  27. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  28. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  29. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  30. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  31. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  32. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  33. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  34. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  35. 2019The Evolved Transformercited by 180
  36. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  37. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  38. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 95; 2025: 90; 2026: 91.

294 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  5. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  6. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  7. 2023A Survey on Model Compression for Large Language Modelscited by 269
  8. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  9. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  11. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  12. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
38 more
  1. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  2. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  3. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  4. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  8. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  9. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  10. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  11. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  12. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  13. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  14. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  15. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  16. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  17. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 81
  18. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  19. 2024Evaluating Quantized Large Language Modelscited by 77
  20. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  21. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  22. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  23. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  24. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  25. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  26. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  27. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  28. 2024The Super Weight in Large Language Modelscited by 52
  29. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  30. 2024A Survey on Transformer Compressioncited by 46
  31. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  32. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  33. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  34. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  35. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  36. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  37. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 39
  38. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38

50 most cited of 294 All 294 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
  2. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
  3. 2025Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
  4. 2023A Speed Odyssey for Deployable Quantization of LLMs
  5. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
  6. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt
  7. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  8. 2025SiLQ: Simple Large Language Model Quantization-Aware Training