# A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions

arXiv: https://arxiv.org/abs/2010.03954 · HTML: https://mlsanity.com/paper/2010.03954/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 30 of 32)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024)
- [Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation](https://mlsanity.com/paper/2305.02231/) (2023)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [Modeling the AC Power Flow Equations with Optimally Compact Neural Networks: Application to Unit Commitment](https://mlsanity.com/paper/2110.11269/) (2021)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [CHISEL: Compression-Aware High-Accuracy Embedded Indoor Localization with Deep Learning](https://mlsanity.com/paper/2107.01192/) (2021)
- [On Neural Network Equivalence Checking using SMT Solvers](https://mlsanity.com/paper/2203.11629/) (2022)
- [Distributed Neural Representation for Reactive in situ Visualization](https://mlsanity.com/paper/2304.10516/) (2023)
- [Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation](https://mlsanity.com/paper/2209.15560/) (2022)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [MD Loss: Efficient Training of 3D Seismic Fault Segmentation Network under Sparse Labels by Weakening Anomaly Annotation](https://mlsanity.com/paper/2110.05319/) (2021)
- [Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization](https://mlsanity.com/paper/2210.09134/) (2022)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [Adversarial Attacks on Machine Learning in Embedded and IoT Platforms](https://mlsanity.com/paper/2303.02214/) (2023)
- [Compression of Recurrent Neural Networks using Matrix Factorization](https://mlsanity.com/paper/2310.12688/) (2023)
- [Order of Compression: A Systematic and Optimal Sequence to Combinationally Compress CNN](https://mlsanity.com/paper/2403.17447/) (2024)
- [Performance optimizations on deep noise suppression models](https://mlsanity.com/paper/2110.04378/) (2021)
- [FCN-Pose: A Pruned and Quantized CNN for Robot Pose Estimation for Constrained Devices](https://mlsanity.com/paper/2205.13272/) (2022)
- [Manifold Regularization for Memory-Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/2305.17119/) (2023)
- [Robustness to distribution shifts of compressed networks for edge devices](https://mlsanity.com/paper/2401.12014/) (2024)
- [Path Signatures and Graph Neural Networks for Slow Earthquake Analysis: Better Together?](https://mlsanity.com/paper/2402.03558/) (2024)
- [A Digital Twin prototype for traffic sign recognition of a learning-enabled autonomous vehicle](https://mlsanity.com/paper/2402.09097/) (2024)
- [Edge-Efficient Deep Learning Models for Automatic Modulation Classification: A Performance Analysis](https://mlsanity.com/paper/2404.15343/) (2024)
- [Adversarial Robustness of Distilled and Pruned Deep Learning-based Wireless Classifiers](https://mlsanity.com/paper/2404.15344/) (2024)
- [Securing the Future: Proactive Threat Hunting for Sustainable IoT Ecosystems](https://mlsanity.com/paper/2406.14804/) (2024)
- [Efficient DNN-Powered Software with Fair Sparse Models](https://mlsanity.com/paper/2407.02805/) (2024)
- [Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization](https://mlsanity.com/paper/2408.00923/) (2024)

## Abstract (arXiv)

Deep Neural Network (DNN) has gained unprecedented performance due to its automated feature extraction capability. This high order performance leads to significant incorporation of DNN models in different Internet of Things (IoT) applications in the past decade. However, the colossal requirement of computation, energy, and storage of DNN models make their deployment prohibitive on resource constraint IoT devices. Therefore, several compression techniques were proposed in recent years for reducing the storage and computation requirements of the DNN model. These techniques on DNN compression have utilized a different perspective for compressing DNN with minimal accuracy compromise. It encourages us to make a comprehensive overview of the DNN compression techniques. In this paper, we present a comprehensive review of existing literature on compressing DNN model that reduces both storage and computation requirements. We divide the existing approaches into five broad categories, i.e., network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous, based upon the mechanism incorporated for compressing the DNN model. The paper also discussed the challenges associated with each category of DNN compression techniques. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.