mlsanityData to 14 Oct 2026

SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization

arXiv 2412.04180 ↗First posted 5 Dec 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or the use of smaller, less capable models. While quantization offers a promising solution utilizing lower precision for model storage, existing methods frequently experience significant performance drops at lower precision levels. Additionally, they typically provide only a limited set of solutions at specific bit levels, many of which are extensively manually tuned. To address these challenges, we propose a new method called SKIM: Scaled K-means clustering wIth Mixed precision. Our approach introduces two novel techniques: 1. A greedy algorithm to solve approximately optimal bit allocation across weight channels, and 2. A trainable scaling vector for non-differentiable K-means clustering. These techniques substantially improve performance and can be adapted to any given bit. Notably, in terms of model perplexity, our method narrows the gap between 3-bit quantized LLaMA models and their full precision counterparts by 16.3% on average.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

33 of 44 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
21 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  3. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  6. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  9. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  10. 2015Coordinate Descent Algorithmscited by 219
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  12. 2024AI and Memory Wallcited by 92
  13. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  14. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  15. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  16. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  17. 2018Deep Neural Network Compression with Single and Multiple Level Quantizationcited by 27
  18. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22
  19. 2021Post-Training Sparsity-Aware Quantizationcited by 17
  20. 2023NUPES : Non-Uniform Post-Training Quantization via Power Exponent Searchcited by 8
  21. 2024decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Pointscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 32
  2. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidthscited by 2
  3. 2025LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillationcited by 1
  4. 2026Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantizationcited by 1
  5. 2026SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Modelscited by 1
  6. 2026QuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
  2. 2024CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs
  3. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospects
  4. 2023eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models
  5. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inference
  6. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
  7. 2025FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs
  8. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs