# Towards Generalized Entropic Sparsification for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2404.04734 · HTML: https://mlsanity.com/paper/2404.04734/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 6 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates](https://mlsanity.com/paper/1907.03141/) (2019)
- [DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation](https://mlsanity.com/paper/2004.02164/) (2020)
- [Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression](https://mlsanity.com/paper/1812.04368/) (2018)
- [An Entropy-based Pruning Method for CNN Compression](https://mlsanity.com/paper/1706.05791/) (2017)
- [A Low Effort Approach to Structured CNN Design Using PCA](https://mlsanity.com/paper/1812.06224/) (2018)
- [ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning](https://mlsanity.com/paper/1905.00136/) (2019)
- [Adversarially Robust Learning via Entropic Regularization](https://mlsanity.com/paper/2008.12338/) (2020)
- [Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks](https://mlsanity.com/paper/2002.09958/) (2020)

## Abstract (arXiv)

Convolutional neural networks (CNNs) are reported to be overparametrized. The search for optimal (minimal) and sufficient architecture is an NP-hard problem as the hyperparameter space for possible network configurations is vast. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint. This allows deploying a numerically scalable algorithm with a sublinear scaling cost. The method is validated on several benchmarks (architectures): (i) MNIST (LeNet) with sparsity 55%-84% and loss in accuracy 0.1%-0.5%, and (ii) CIFAR-10 (VGG-16, ResNet18) with sparsity 73-89% and loss in accuracy 0.1%-0.5%.