# A Survey of Model Compression and Acceleration for Deep Neural Networks

arXiv: https://arxiv.org/abs/1710.09282 · HTML: https://mlsanity.com/paper/1710.09282/

## Key facts

- Cited by 301 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2017.

## Built on by (most cited first, 30 of 301)

- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [A Unified Lottery Ticket Hypothesis for Graph Neural Networks](https://mlsanity.com/paper/2102.06790/) (2021)
- [Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference](https://mlsanity.com/paper/2110.03742/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [COIN++: Neural Compression Across Modalities](https://mlsanity.com/paper/2201.12904/) (2022)
- [PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization](https://mlsanity.com/paper/2211.13609/) (2022)
- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization](https://mlsanity.com/paper/2110.06554/) (2021)
- [Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation](https://mlsanity.com/paper/2305.02231/) (2023)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2101.06871/) (2021)
- [Meta Faster R-CNN: Towards Accurate Few-Shot Object Detection with Attentive Feature Alignment](https://mlsanity.com/paper/2104.07719/) (2021)
- [CDFI: Compression-Driven Network Design for Frame Interpolation](https://mlsanity.com/paper/2103.10559/) (2021)
- [MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer](https://mlsanity.com/paper/2403.02991/) (2024)
- [ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models](https://mlsanity.com/paper/2402.13516/) (2024)
- [Blueprint Separable Residual Network for Efficient Image Super-Resolution](https://mlsanity.com/paper/2205.05996/) (2022)
- [Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2207.05273/) (2022)
- [Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression](https://mlsanity.com/paper/2109.03228/) (2021)
- [Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains](https://mlsanity.com/paper/2106.13474/) (2021)

## Abstract (arXiv)

Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.