# Effective Interplay between Sparsity and Quantization: From Theory to Practice

arXiv: https://arxiv.org/abs/2405.20935 · HTML: https://mlsanity.com/paper/2405.20935/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 87 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)

## Built on by (most cited first, 19 of 19)

- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression](https://mlsanity.com/paper/2502.16638/) (2025)
- [SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression](https://mlsanity.com/paper/2410.09615/) (2024)
- [When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models](https://mlsanity.com/paper/2406.07368/) (2024)
- [Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference](https://mlsanity.com/paper/2505.22913/) (2025)
- [Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs](https://mlsanity.com/paper/2509.11177/) (2025)
- [Unified Scaling Laws for Compressed Representations](https://mlsanity.com/paper/2506.01863/) (2025)
- [Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression](https://mlsanity.com/paper/2603.18426/) (2026)
- [Communication Compression for Distributed Learning without Control Variates](https://mlsanity.com/paper/2412.04538/) (2024)
- [DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model](https://mlsanity.com/paper/2505.19349/) (2025)
- [SwiftPrune: Hessian-Free Weight Pruning for Large Language Models](https://mlsanity.com/paper/2501.16376/) (2025)
- [Efficient Quantization-Aware Neural Receivers: Beyond Post-Training Quantization](https://mlsanity.com/paper/2509.13786/) (2025)
- [Cat: Post-Training Quantization Error Reduction via Cluster-based Affine Transformation](https://mlsanity.com/paper/2509.26277/) (2025)
- [Extreme Model Compression with Structured Sparsity at Low Precision](https://mlsanity.com/paper/2511.08360/) (2025)
- [Communication Compression for Distributed Learning with Aggregate and Server-Guided Feedback](https://mlsanity.com/paper/2512.22623/) (2025)
- [Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression](https://mlsanity.com/paper/2606.07819/) (2026)
- [Reliability Scaling Laws for Quantized Large Language Models](https://mlsanity.com/paper/2607.10855/) (2026)
- [Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts](https://mlsanity.com/paper/2610.02241/) (2026)

## Abstract (arXiv)

The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two prominent compression methods that have been shown to reduce DNNs' computational and memory footprints significantly while preserving model accuracy. However, how these two methods interact when combined together remains a key question for developers, as many tacitly assume that they are orthogonal, meaning that their combined use does not introduce additional errors beyond those introduced by each method independently. In this paper, we provide the first mathematical proof that sparsity and quantization are non-orthogonal. We corroborate these results with experiments spanning a range of large language models, including the OPT and LLaMA model families (with 125M to 8B parameters), and vision models like ViT and ResNet. We show that the order in which we apply these methods matters because applying quantization before sparsity may disrupt the relative importance of tensor elements, which may inadvertently remove significant elements from a tensor. More importantly, we show that even if applied in the correct order, the compounded errors from sparsity and quantization can significantly harm accuracy. Our findings extend to the efficient deployment of large models in resource-constrained compute platforms to reduce serving cost, offering insights into best practices for applying these compression methods to maximize hardware resource efficiency without compromising accuracy.