mlsanityData to 7 Oct 2026

SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

arXiv 2306.03078 ↗First posted 5 Jun 2023cs.CL cs.LGcited by 359

Not on the map. Why?

Cited by 44 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities. By compressing such LLMs via quantization to 3-4 bits per parameter, they can fit into memory-limited devices such as laptops and mobile phones, enabling personalized use. However, quantization down to 3-4 bits per parameter usually leads to moderate-to-high accuracy losses, especially for smaller models in the 1-10B parameter range, which are well-suited for edge deployments. To address this accuracy issue, we introduce the Sparse-Quantized Representation (SpQR), a new compressed format and quantization technique which enables for the first time near-lossless compression of LLMs across model scales, while reaching similar compression levels to previous methods. SpQR works by identifying and isolating outlier weights, which cause particularly-large quantization errors, and storing them in higher precision, while compressing all other weights to 3-4 bits, and achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B parameter LLM on a single 24 GB consumer GPU without any performance degradation at 15% speedup thus making powerful LLMs available to consumer without any downsides. SpQR comes with efficient algorithms for both encoding weights into its format, as well as decoding them efficiently at runtime. Specifically, we provide an efficient GPU inference algorithm for SpQR which yields faster inference than 16-bit baselines at similar accuracy, while enabling memory compression gains of more than 4x.

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

35 of 61 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
23 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  13. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  14. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  15. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  16. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  17. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  18. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  19. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  20. 2019MLPerf Inference Benchmarkcited by 195
  21. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  22. 2023Intriguing Properties of Quantization at Scalecited by 48
  23. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 129; 2025: 98; 2026: 95.

359 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  5. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  6. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  8. 2023A Survey on Model Compression for Large Language Modelscited by 263
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  12. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
38 more
  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  4. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  9. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  10. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  11. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  12. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  13. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  14. 2024Scaling Laws for Precisioncited by 92
  15. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  16. 2024AI and Memory Wallcited by 89
  17. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  18. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  19. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  20. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  21. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  22. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  23. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  24. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  25. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  26. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  27. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  28. 2024Exploiting LLM Quantizationcited by 61
  29. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  30. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  31. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  32. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  33. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  34. 2024The Super Weight in Large Language Modelscited by 51
  35. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  36. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  37. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  38. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47

50 most cited of 359 All 359 (JSON)