# Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction

arXiv: https://arxiv.org/abs/2210.07451 · HTML: https://mlsanity.com/paper/2210.07451/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Fixed Point Quantization of Deep Convolutional Networks](https://mlsanity.com/paper/1511.06393/) (2015)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Leveraging Image Complexity in Macro-Level Neural Network Design for Medical Image Segmentation](https://mlsanity.com/paper/2112.11065/) (2021)
- [RePr: Improved Training of Convolutional Filters](https://mlsanity.com/paper/1811.07275/) (2018)
- [RC-Net: A Convolutional Neural Network for Retinal Vessel Segmentation](https://mlsanity.com/paper/2112.11078/) (2021)
- [A Derivative-free Method for Quantum Perceptron Training in Multi-layered Neural Networks](https://mlsanity.com/paper/2009.13264/) (2020)

## Built on by (most cited first, 5 of 5)

- [TBConvL-Net: A Hybrid Deep Learning Architecture for Robust Medical Image Segmentation](https://mlsanity.com/paper/2409.03367/) (2024)
- [LDMRes-Net: Enabling Efficient Medical Image Segmentation on IoT and Edge Platforms](https://mlsanity.com/paper/2306.06145/) (2023)
- [AD-Net: Attention-based dilated convolutional residual network with guided decoder for robust skin lesion segmentation](https://mlsanity.com/paper/2409.05420/) (2024)
- [Progressive Class-Wise Attention (PCA) Approach for Diagnosing Skin Lesions](https://mlsanity.com/paper/2306.07300/) (2023)
- [EDDense-Net: Fully Dense Encoder Decoder Network for Joint Segmentation of Optic Cup and Disc](https://mlsanity.com/paper/2308.10192/) (2023)

## Abstract (arXiv)

Compression of convolutional neural network models has recently been dominated by pruning approaches. A class of previous works focuses solely on pruning the unimportant filters to achieve network compression. Another important direction is the design of sparsity-inducing constraints which has also been explored in isolation. This paper presents a novel training scheme based on composite constraints that prune redundant filters and minimize their effect on overall network learning via sparsity promotion. Also, as opposed to prior works that employ pseudo-norm-based sparsity-inducing constraints, we propose a sparse scheme based on gradient counting in our framework. Our tests on several pixel-wise segmentation benchmarks show that the number of neurons and the memory footprint of networks in the test phase are significantly reduced without affecting performance. MobileNetV3 and UNet, two well-known architectures, are used to test the proposed scheme. Our network compression method not only results in reduced parameters but also achieves improved performance compared to MobileNetv3, which is an already optimized architecture.