# An Overview of Neural Network Compression

arXiv: https://arxiv.org/abs/2006.03669 · HTML: https://mlsanity.com/paper/2006.03669/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2020.

## Built on by (most cited first, 30 of 59)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Version](https://mlsanity.com/paper/2302.12721/) (2023)
- [Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks](https://mlsanity.com/paper/2106.03795/) (2021)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [Post-training Quantization for Neural Networks with Provable Guarantees](https://mlsanity.com/paper/2201.11113/) (2022)
- [VORTA: Efficient Video Diffusion via Routing Sparse Attention](https://mlsanity.com/paper/2505.18809/) (2025)
- [Improving Monocular Visual Odometry Using Learned Depth](https://mlsanity.com/paper/2204.01268/) (2022)
- [On Neural Network Equivalence Checking using SMT Solvers](https://mlsanity.com/paper/2203.11629/) (2022)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Implicit-explicit Integrated Representations for Multi-view Video Compression](https://mlsanity.com/paper/2311.17350/) (2023)
- [Quantum-Enhanced Parameter-Efficient Learning for Typhoon Trajectory Forecasting](https://mlsanity.com/paper/2505.09395/) (2025)
- [Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD](https://mlsanity.com/paper/2306.08125/) (2023)
- [Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing](https://mlsanity.com/paper/2403.04759/) (2024)
- [Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features](https://mlsanity.com/paper/2410.02237/) (2024)
- [Auto-Compressing Subset Pruning for Semantic Image Segmentation](https://mlsanity.com/paper/2201.11103/) (2022)
- [Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think](https://mlsanity.com/paper/2211.17180/) (2022)
- [Machine Learning and the Future of Bayesian Computation](https://mlsanity.com/paper/2304.11251/) (2023)
- [Reconstruct the Pruned Model without Any Retraining](https://mlsanity.com/paper/2407.13331/) (2024)
- [Divide-and-Conquer the NAS puzzle in Resource Constrained Federated Learning Systems](https://mlsanity.com/paper/2305.07135/) (2023)
- [Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping](https://mlsanity.com/paper/2308.07641/) (2023)
- [Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks](https://mlsanity.com/paper/2502.02766/) (2025)
- [Distributed Quantum Neural Networks on Distributed Photonic Quantum Computing](https://mlsanity.com/paper/2505.08474/) (2025)
- [Riemannian Optimization on Tree Tensor Networks with Application in Machine Learning](https://mlsanity.com/paper/2507.21726/) (2025)
- [Structured Bayesian Compression for Deep Neural Networks Based on The Turbo-VBI Approach](https://mlsanity.com/paper/2302.10483/) (2023)
- [Pruning at Initialization -- A Sketching Perspective](https://mlsanity.com/paper/2305.17559/) (2023)
- [Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models](https://mlsanity.com/paper/2307.05972/) (2023)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [Dendron: Enhancing Human Activity Recognition with On-Device TinyML Learning](https://mlsanity.com/paper/2503.01353/) (2025)
- [On the Interaction of Compressibility and Adversarial Robustness](https://mlsanity.com/paper/2507.17725/) (2025)

## Abstract (arXiv)

Overparameterized networks trained to convergence have shown impressive performance in domains such as computer vision and natural language processing. Pushing state of the art on salient tasks within these domains corresponds to these models becoming larger and more difficult for machine learning practitioners to use given the increasing memory and storage requirements, not to mention the larger carbon footprint. Thus, in recent years there has been a resurgence in model compression techniques, particularly for deep convolutional neural networks and self-attention based networks such as the Transformer. Hence, this paper provides a timely overview of both old and current compression techniques for deep neural networks, including pruning, quantization, tensor decomposition, knowledge distillation and combinations thereof. We assume a basic familiarity with deep learning architecturesfootnoteFor an introduction to deep learning, see ~citetgoodfellow2016deep, namely, Recurrent Neural Networks~citep[(RNNs)][]rumelhart1985learning,hochreiter1997long, Convolutional Neural Networks~citepfukushima1980neocognitron~footnoteFor an up to date overview see~citetkhan2019survey and Self-Attention based networks~citepvaswani2017attentionfootnoteFor a general overview of self-attention networks, see ~citetchaudhari2019attentive.,footnoteFor more detail and their use in natural language processing, see~citethu2019introductory. Most of the papers discussed are proposed in the context of at least one of these DNN architectures.