# Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units

arXiv: https://arxiv.org/abs/1704.07724 · HTML: https://mlsanity.com/paper/1704.07724/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2017.

## Built on by (most cited first, 6 of 6)

- [DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models](https://mlsanity.com/paper/2402.19481/) (2024)
- [Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models](https://mlsanity.com/paper/2211.02048/) (2022)
- [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021)
- [PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization](https://mlsanity.com/paper/2103.01548/) (2021)
- [Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference](https://mlsanity.com/paper/2305.17423/) (2023)
- [High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks](https://mlsanity.com/paper/2104.08314/) (2021)

## Abstract (arXiv)

Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks. An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons. In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures. And then we use the sparsity property of ReLUs to accelerate the calculation of convolution by skipping calculations of zero-valued neurons. The proposed sparse convolution algorithm achieves some speedup improvements on CPUs compared to the traditional matrix-matrix multiplication algorithm for convolution when the sparsity is not less than 0.9.