mlsanityData to 14 Oct 2026

Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs

arXiv 2405.14597 ↗First posted 23 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce Integer Scale, a novel post-training quantization scheme for large language models that effectively resolves the inference bottleneck in current fine-grained quantization approaches while maintaining similar accuracies. Integer Scale is a free lunch as it requires no extra calibration or fine-tuning which will otherwise incur additional costs. It can be used plug-and-play for most fine-grained quantization methods. Its integration results in at most 1.85x end-to-end speed boost over the original counterpart with comparable accuracy. Additionally, due to the orchestration of the proposed Integer Scale and fine-grained quantization, we resolved the quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible performance degradation, and it comes with an end-to-end speed boost of 2.13x, and 2.31x compared with their FP16 versions respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
28 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2024Mixtral of Expertscited by 1,823
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2021A White Paper on Neural Network Quantizationcited by 472
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  13. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  14. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  15. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  16. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  17. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  18. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  19. 2022FP8 Formats for Deep Learningcited by 196
  20. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  21. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  22. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  23. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  24. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  25. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  26. 2024FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designcited by 12
  27. 2023A Speed Odyssey for Deployable Quantization of LLMscited by 10
  28. 2023Dual Grained Quantization: Efficient Fine-Grained Quantization for LLMcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
  2. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  3. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
  4. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  5. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
  6. 2024Post Training Quantization of Large Language Models with Microscaling Formats
  7. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  8. 2025FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs