mlsanityData to 7 Oct 2026

Towards Generalized Entropic Sparsification for Convolutional Neural Networks

arXiv 2404.04734 ↗First posted 6 Apr 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Convolutional neural networks (CNNs) are reported to be overparametrized. The search for optimal (minimal) and sufficient architecture is an NP-hard problem as the hyperparameter space for possible network configurations is vast. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint. This allows deploying a numerically scalable algorithm with a sublinear scaling cost. The method is validated on several benchmarks (architectures): (i) MNIST (LeNet) with sparsity 55%-84% and loss in accuracy 0.1%-0.5%, and (ii) CIFAR-10 (VGG-16, ResNet18) with sparsity 73-89% and loss in accuracy 0.1%-0.5%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

22 of 40 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  7. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  8. 2018Rethinking the Value of Network Pruningcited by 595
  9. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  10. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  11. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  12. 2013Predicting Parameters in Deep Learningcited by 223
10 more
  1. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  2. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  3. 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
  4. 2020DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocationcited by 47
  5. 2018Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compressioncited by 40
  6. 2017An Entropy-based Pruning Method for CNN Compressioncited by 37
  7. 2018A Low Effort Approach to Structured CNN Design Using PCAcited by 14
  8. 2019ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruningcited by 13
  9. 2020Adversarially Robust Learning via Entropic Regularizationcited by 9
  10. 2020Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networkscited by 6

Built on by

Not cited yet.