# Compression Scaling Laws:Unifying Sparsity and Quantization

arXiv: https://arxiv.org/abs/2502.16440 · HTML: https://mlsanity.com/paper/2502.16440/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 14 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [Scaling Laws for Sparsely-Connected Foundation Models](https://mlsanity.com/paper/2309.08520/) (2023)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)

## Built on by (most cited first, 11 of 11)

- [Quartet: Native FP4 Training Can Be Optimal for Large Language Models](https://mlsanity.com/paper/2505.14669/) (2025)
- [INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats](https://mlsanity.com/paper/2510.25602/) (2025)
- [Scaling Law for Quantization-Aware Training](https://mlsanity.com/paper/2505.14302/) (2025)
- [LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts](https://mlsanity.com/paper/2601.18089/) (2026)
- [Unified Scaling Laws for Compressed Representations](https://mlsanity.com/paper/2506.01863/) (2025)
- [Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory](https://mlsanity.com/paper/2510.12077/) (2025)
- [From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction](https://mlsanity.com/paper/2507.03052/) (2025)
- [GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation](https://mlsanity.com/paper/2606.01412/) (2026)
- [Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding](https://mlsanity.com/paper/2505.18758/) (2025)
- [Reliability Scaling Laws for Quantized Large Language Models](https://mlsanity.com/paper/2607.10855/) (2026)
- [Mixed Precision Scaling Laws for Quantized Pretraining](https://mlsanity.com/paper/2610.17637/) (2026)

## Abstract (arXiv)

We investigate how different compression techniques -- such as weight and activation quantization, and weight sparsity -- affect the scaling behavior of large language models (LLMs) during pretraining. Building on previous work showing that weight sparsity acts as a constant multiplier on model size in scaling laws, we demonstrate that this "effective parameter" scaling pattern extends to quantization as well. Specifically, we establish that weight-only quantization achieves strong parameter efficiency multipliers, while full quantization of both weights and activations shows diminishing returns at lower bitwidths. Our results suggest that different compression techniques can be unified under a common scaling law framework, enabling principled comparison and combination of these methods.