mlsanityData to 14 Oct 2026

LLM Compression: How Far Can We Go in Balancing Size and Performance?

arXiv 2508.11318 ↗First posted 15 Aug 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantization is an essential and popular technique for improving the accessibility of large language models (LLMs) by reducing memory usage and computational costs while maintaining performance. In this study, we apply 4-bit Group Scaling Quantization (GSQ) and Generative Pretrained Transformer Quantization (GPTQ) to LLaMA 1B, Qwen 0.5B, and PHI 1.5B, evaluating their impact across multiple NLP tasks. We benchmark these models on MS MARCO (Information Retrieval), BoolQ (Boolean Question Answering), and GSM8K (Mathematical Reasoning) datasets, assessing both accuracy and efficiency across various tasks. The study measures the trade-offs between model compression and task performance, analyzing key evaluation metrics, namely accuracy, inference latency, and throughput (total output tokens generated per second), providing insights into the suitability of low-bit quantization for real-world deployment. Using the results, users can then make suitable decisions based on the specifications that need to be met. We discuss the pros and cons of GSQ and GPTQ techniques on models of different sizes, which also serve as a benchmark for future experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 17 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

9 of 17 references.

References · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  5. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  6. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  7. 2024Post Training Quantization of Large Language Models with Microscaling Formatscited by 12
  8. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniquescited by 10
  9. 2024GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inferencecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  2. 2024GPTQT: Quantize Large Language Models Twice to Push the Efficiency
  3. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  4. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis
  5. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  6. 2026Two-Stage Grid Optimization for Group-wise Quantization of LLMs
  7. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
  8. 2024CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization