# The Power of Sparsity in Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1702.06257 · HTML: https://mlsanity.com/paper/1702.06257/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2017.

## Built on by (most cited first, 2 of 2)

- [Differentiable Transportation Pruning](https://mlsanity.com/paper/2307.08483/) (2023)
- [Training of deep residual networks with stochastic MG/OPT](https://mlsanity.com/paper/2108.04052/) (2021)

## Abstract (arXiv)

Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.