# Post-Training Sparsity-Aware Quantization

arXiv: https://arxiv.org/abs/2105.11010 · HTML: https://mlsanity.com/paper/2105.11010/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 23 May 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Low-bit Quantization of Neural Networks for Efficient Inference](https://mlsanity.com/paper/1902.06822/) (2019)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [NVIDIA Tensor Core Programmability, Performance & Precision](https://mlsanity.com/paper/1803.04014/) (2018)
- [Post-Training Piecewise Linear Quantization for Deep Neural Networks](https://mlsanity.com/paper/2002.00104/) (2020)
- [Robust Quantization: One Model to Rule Them All](https://mlsanity.com/paper/2002.07686/) (2020)
- [Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy](https://mlsanity.com/paper/2103.03014/) (2021)
- [Value-aware Quantization for Training and Inference of Neural Networks](https://mlsanity.com/paper/1804.07802/) (2018)
- [Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking](https://mlsanity.com/paper/1804.06826/) (2018)
- [Parallel Multi Channel Convolution using General Matrix Multiplication](https://mlsanity.com/paper/1704.04428/) (2017)
- [Modeling Deep Learning Accelerator Enabled GPUs](https://mlsanity.com/paper/1811.08309/) (2018)
- [Thanks for Nothing: Predicting Zero-Valued Activations with Lightweight Convolutional Neural Networks](https://mlsanity.com/paper/1909.07636/) (2019)
- [Post-Training BatchNorm Recalibration](https://mlsanity.com/paper/2010.05625/) (2020)
- [Non-Blocking Simultaneous Multithreading: Embracing the Resiliency of Deep Neural Networks](https://mlsanity.com/paper/2004.09309/) (2020)
- [Sparse Systolic Tensor Array for Efficient CNN Hardware Acceleration](https://mlsanity.com/paper/2009.02381/) (2020)

## Built on by (most cited first, 17 of 17)

- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Boost Vision Transformer with GPU-Friendly Sparsity and Quantization](https://mlsanity.com/paper/2305.10727/) (2023)
- [Extreme Compression for Pre-trained Transformers Made Simple and Efficient](https://mlsanity.com/paper/2206.01859/) (2022)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [TSPTQ-ViT: Two-scaled post-training quantization for vision transformer](https://mlsanity.com/paper/2305.12901/) (2023)
- [SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization](https://mlsanity.com/paper/2412.04180/) (2024)
- [Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition](https://mlsanity.com/paper/2206.15408/) (2022)
- [Efficient Adaptive Activation Rounding for Post-Training Quantization](https://mlsanity.com/paper/2208.11945/) (2022)
- [QFT: Post-training quantization via fast joint finetuning of all degrees of freedom](https://mlsanity.com/paper/2212.02634/) (2022)
- [Redistribution of Weights and Activations for AdderNet Quantization](https://mlsanity.com/paper/2212.10200/) (2022)
- [On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2403.17154/) (2024)
- [AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models](https://mlsanity.com/paper/2510.20348/) (2025)
- [Characterizing and Understanding the Behavior of Quantized Models for Reliable Deployment](https://mlsanity.com/paper/2204.04220/) (2022)
- [Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview](https://mlsanity.com/paper/2409.11650/) (2024)
- [DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation](https://mlsanity.com/paper/2409.14307/) (2024)
- [On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2411.00907/) (2024)
- [BEExformer: A Fast Inferencing Binarized Transformer with Early Exits](https://mlsanity.com/paper/2412.05225/) (2024)

## Abstract (arXiv)

Quantization is a technique used in deep neural networks (DNNs) to increase execution performance and hardware efficiency. Uniform post-training quantization (PTQ) methods are common, since they can be implemented efficiently in hardware and do not require extensive hardware resources or a training set. Mapping FP32 models to INT8 using uniform PTQ yields models with negligible accuracy degradation; however, reducing precision below 8 bits with PTQ is challenging, as accuracy degradation becomes noticeable, due to the increase in quantization noise. In this paper, we propose a sparsity-aware quantization (SPARQ) method, in which the unstructured and dynamic activation sparsity is leveraged in different representation granularities. 4-bit quantization, for example, is employed by dynamically examining the bits of 8-bit values and choosing a window of 4 bits, while first skipping zero-value bits. Moreover, instead of quantizing activation-by-activation to 4 bits, we focus on pairs of 8-bit activations and examine whether one of the two is equal to zero. If one is equal to zero, the second can opportunistically use the other's 4-bit budget; if both do not equal zero, then each is dynamically quantized to 4 bits, as described. SPARQ achieves minor accuracy degradation and a practical hardware implementation. The code is available at https://github.com/gilshm/sparq.