# Pruning and Quantization for Deep Neural Network Acceleration: A Survey

arXiv: https://arxiv.org/abs/2101.09671 · HTML: https://mlsanity.com/paper/2101.09671/

## Key facts

- Cited by 263 papers in the map (data through 14 Oct 2026).
- Builds on 153 of its 297 references that are in the map.
- First posted on arXiv 24 Jan 2021.

## Builds on (most cited first, 30 of 153)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)

## Built on by (most cited first, 30 of 263)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)
- [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023)
- [COIN++: Neural Compression Across Modalities](https://mlsanity.com/paper/2201.12904/) (2022)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning](https://mlsanity.com/paper/2402.03666/) (2024)
- [An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry](https://mlsanity.com/paper/2303.02552/) (2023)
- [Causal Recurrent Variational Autoencoder for Medical Time Series Generation](https://mlsanity.com/paper/2301.06574/) (2023)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)
- [LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression](https://mlsanity.com/paper/2309.14021/) (2023)
- [Deep Learning for Iris Recognition: A Survey](https://mlsanity.com/paper/2210.05866/) (2022)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [Quantization without Tears](https://mlsanity.com/paper/2411.13918/) (2024)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques](https://mlsanity.com/paper/2406.02500/) (2024)
- [Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies](https://mlsanity.com/paper/2503.02891/) (2025)
- [FS-Real: Towards Real-World Cross-Device Federated Learning](https://mlsanity.com/paper/2303.13363/) (2023)
- [Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation](https://mlsanity.com/paper/2404.01365/) (2024)
- [Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models](https://mlsanity.com/paper/2412.11041/) (2024)

## Abstract (arXiv)

Deep neural networks have been applied in many applications exhibiting extraordinary abilities in the field of computer vision. However, complex network architectures challenge efficient real-time deployment and require significant computation resources and energy costs. These challenges can be overcome through optimizations such as network compression. Network compression can often be realized with little loss of accuracy. In some cases accuracy may even improve. This paper provides a survey on two types of network compression: pruning and quantization. Pruning can be categorized as static if it is performed offline or dynamic if it is performed at run-time. We compare pruning techniques and describe criteria used to remove redundant computations. We discuss trade-offs in element-wise, channel-wise, shape-wise, filter-wise, layer-wise and even network-wise pruning. Quantization reduces computations by reducing the precision of the datatype. Weights, biases, and activations may be quantized typically to 8-bit integers although lower bit width implementations are also discussed including binary neural networks. Both pruning and quantization can be used independently or combined. We compare current techniques, analyze their strengths and weaknesses, present compressed network accuracy results on a number of frameworks, and provide practical guidance for compressing networks.