# LLM Compression: How Far Can We Go in Balancing Size and Performance?

arXiv: https://arxiv.org/abs/2508.11318 · HTML: https://mlsanity.com/paper/2508.11318/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Builds on (most cited first, 9 of 9)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs](https://mlsanity.com/paper/2308.09723/) (2023)
- [Post Training Quantization of Large Language Models with Microscaling Formats](https://mlsanity.com/paper/2405.07135/) (2024)
- [Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques](https://mlsanity.com/paper/2411.06084/) (2024)
- [GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2412.17560/) (2024)

## Abstract (arXiv)

Quantization is an essential and popular technique for improving the accessibility of large language models (LLMs) by reducing memory usage and computational costs while maintaining performance. In this study, we apply 4-bit Group Scaling Quantization (GSQ) and Generative Pretrained Transformer Quantization (GPTQ) to LLaMA 1B, Qwen 0.5B, and PHI 1.5B, evaluating their impact across multiple NLP tasks. We benchmark these models on MS MARCO (Information Retrieval), BoolQ (Boolean Question Answering), and GSM8K (Mathematical Reasoning) datasets, assessing both accuracy and efficiency across various tasks. The study measures the trade-offs between model compression and task performance, analyzing key evaluation metrics, namely accuracy, inference latency, and throughput (total output tokens generated per second), providing insights into the suitability of low-bit quantization for real-world deployment. Using the results, users can then make suitable decisions based on the specifications that need to be met. We discuss the pros and cons of GSQ and GPTQ techniques on models of different sizes, which also serve as a benchmark for future experiments.