mlsanityData to 7 Oct 2026

On the Compression of Neural Networks Using ℓ0-Norm Regularization and Weight Pruning

arXiv 2109.05075 ↗First posted 10 Sep 2021cs.LG cs.AI eess.SPcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite the growing availability of high-capacity computational platforms, implementation complexity still has been a great concern for the real-world deployment of neural networks. This concern is not exclusively due to the huge costs of state-of-the-art network architectures, but also due to the recent push towards edge intelligence and the use of neural networks in embedded applications. In this context, network compression techniques have been gaining interest due to their ability for reducing deployment costs while keeping inference accuracy at satisfactory levels. The present paper is dedicated to the development of a novel compression scheme for neural networks. To this end, a new form of ℓ0-norm-based regularization is firstly developed, which is capable of inducing strong sparseness in the network during training. Then, targeting the smaller weights of the trained network with pruning techniques, smaller yet highly effective networks can be obtained. The proposed compression scheme also involves the use of ℓ2-norm regularization to avoid overfitting as well as fine tuning to improve the performance of the pruned network. Experimental results are presented aiming to show the effectiveness of the proposed scheme as well as to make comparisons with competing approaches.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 63 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

27 of 63 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2020What is the State of Neural Network Pruning?cited by 578
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  8. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  9. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  10. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  11. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  12. 2016Dynamic Network Surgery for Efficient DNNscited by 233
15 more
  1. 2013Predicting Parameters in Deep Learningcited by 223
  2. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  3. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
  4. 2017Bayesian Compression for Deep Learningcited by 116
  5. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  6. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  7. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  8. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  9. 2020DHP: Differentiable Meta Pruning via HyperNetworkscited by 57
  10. 2020Methods for Pruning Deep Neural Networkscited by 49
  11. 2014Efficient and Accurate Approximations of Nonlinear Convolutional Networkscited by 30
  12. 2019Play and Prune: Adaptive Filter Pruning for Deep Model Compressioncited by 15
  13. 2016DropNeuron: Simplifying the Structure of Deep Neural Networkscited by 11
  14. 2019L0-ARM: Network Sparsification via Stochastic Binary Optimizationcited by 7
  15. 2015Towards Structured Deep Neural Network for Automatic Speech Recognitioncited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 3.

4 papers.

  1. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  2. 2025Efficient compression of neural networks and datasetscited by 1
  3. 2025Probabilistic and nonlinear compressive sensingcited by 0
  4. 2025GoPrune: Accelerated Structured Pruning with ℓ2,p-Norm Optimizationcited by 0