# Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2001.10710 · HTML: https://mlsanity.com/paper/2001.10710/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2020.

## Built on by (most cited first, 15 of 15)

- [HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise](https://mlsanity.com/paper/2110.11417/) (2021)
- [Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints](https://mlsanity.com/paper/2208.04425/) (2022)
- [Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference](https://mlsanity.com/paper/2301.09254/) (2023)
- [P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications](https://mlsanity.com/paper/2203.04737/) (2022)
- [Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge](https://mlsanity.com/paper/2208.13092/) (2022)
- [ViTA: A Vision Transformer Inference Accelerator for Edge Applications](https://mlsanity.com/paper/2302.09108/) (2023)
- [Personalized Semi-Supervised Federated Learning for Human Activity Recognition](https://mlsanity.com/paper/2104.08094/) (2021)
- [Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design](https://mlsanity.com/paper/2309.13015/) (2023)
- [Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNs](https://mlsanity.com/paper/2302.03523/) (2022)
- [Block Selective Reprogramming for On-device Training of Vision Transformers](https://mlsanity.com/paper/2405.10951/) (2024)
- [High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks](https://mlsanity.com/paper/2104.08314/) (2021)
- [Artificial Neural Networks generated by Low Discrepancy Sequences](https://mlsanity.com/paper/2103.03543/) (2021)
- [Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators](https://mlsanity.com/paper/2207.00068/) (2022)
- [STen: Productive and Efficient Sparsity in PyTorch](https://mlsanity.com/paper/2304.07613/) (2023)
- [Approximation Capabilities of Neural Networks using Morphological Perceptrons and Generalizations](https://mlsanity.com/paper/2207.07832/) (2022)

## Abstract (arXiv)

The high energy cost of processing deep convolutional neural networks impedes their ubiquitous deployment in energy-constrained platforms such as embedded systems and IoT devices. This work introduces convolutional layers with pre-defined sparse 2D kernels that have support sets that repeat periodically within and across filters. Due to the efficient storage of our periodic sparse kernels, the parameter savings can translate into considerable improvements in energy efficiency due to reduced DRAM accesses, thus promising significant improvements in the trade-off between energy consumption and accuracy for both training and inference. To evaluate this approach, we performed experiments with two widely accepted datasets, CIFAR-10 and Tiny ImageNet in sparse variants of the ResNet18 and VGG16 architectures. Compared to baseline models, our proposed sparse variants require up to 82% fewer model parameters with 5.6times fewer FLOPs with negligible loss in accuracy for ResNet18 on CIFAR-10. For VGG16 trained on Tiny ImageNet, our approach requires 5.8times fewer FLOPs and up to 83.3% fewer model parameters with a drop in top-5 (top-1) accuracy of only 1.2% (2.1%). We also compared the performance of our proposed architectures with that of ShuffleNet andMobileNetV2. Using similar hyperparameters and FLOPs, our ResNet18 variants yield an average accuracy improvement of 2.8%.