# On the Compression of Neural Networks Using ℓ0-Norm Regularization and Weight Pruning

arXiv: https://arxiv.org/abs/2109.05075 · HTML: https://mlsanity.com/paper/2109.05075/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 63 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Data-Driven Sparse Structure Selection for Deep Neural Networks](https://mlsanity.com/paper/1707.01213/) (2017)
- [A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1804.03294/) (2018)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Data-free parameter pruning for Deep Neural Networks](https://mlsanity.com/paper/1507.06149/) (2015)
- [Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers](https://mlsanity.com/paper/1802.00124/) (2018)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Global Sparse Momentum SGD for Pruning Very Deep Neural Networks](https://mlsanity.com/paper/1909.12778/) (2019)
- [DHP: Differentiable Meta Pruning via HyperNetworks](https://mlsanity.com/paper/2003.13683/) (2020)
- [Methods for Pruning Deep Neural Networks](https://mlsanity.com/paper/2011.00241/) (2020)
- [Efficient and Accurate Approximations of Nonlinear Convolutional Networks](https://mlsanity.com/paper/1411.4229/) (2014)
- [Play and Prune: Adaptive Filter Pruning for Deep Model Compression](https://mlsanity.com/paper/1905.04446/) (2019)
- [DropNeuron: Simplifying the Structure of Deep Neural Networks](https://mlsanity.com/paper/1606.07326/) (2016)
- [L0-ARM: Network Sparsification via Stochastic Binary Optimization](https://mlsanity.com/paper/1904.04432/) (2019)
- [Towards Structured Deep Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/1511.02506/) (2015)

## Built on by (most cited first, 4 of 4)

- [CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning](https://mlsanity.com/paper/2310.12972/) (2023)
- [Efficient compression of neural networks and datasets](https://mlsanity.com/paper/2505.17469/) (2025)
- [Probabilistic and nonlinear compressive sensing](https://mlsanity.com/paper/2509.15060/) (2025)
- [GoPrune: Accelerated Structured Pruning with ℓ2,p-Norm Optimization](https://mlsanity.com/paper/2511.22120/) (2025)

## Abstract (arXiv)

Despite the growing availability of high-capacity computational platforms, implementation complexity still has been a great concern for the real-world deployment of neural networks. This concern is not exclusively due to the huge costs of state-of-the-art network architectures, but also due to the recent push towards edge intelligence and the use of neural networks in embedded applications. In this context, network compression techniques have been gaining interest due to their ability for reducing deployment costs while keeping inference accuracy at satisfactory levels. The present paper is dedicated to the development of a novel compression scheme for neural networks. To this end, a new form of ℓ0-norm-based regularization is firstly developed, which is capable of inducing strong sparseness in the network during training. Then, targeting the smaller weights of the trained network with pruning techniques, smaller yet highly effective networks can be obtained. The proposed compression scheme also involves the use of ℓ2-norm regularization to avoid overfitting as well as fine tuning to improve the performance of the pruned network. Experimental results are presented aiming to show the effectiveness of the proposed scheme as well as to make comparisons with competing approaches.