# Sparse Activity and Sparse Connectivity in Supervised Learning

arXiv: https://arxiv.org/abs/1603.08367 · HTML: https://mlsanity.com/paper/1603.08367/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2016.

## Built on by (most cited first, 3 of 3)

- [Sparse Super-Regular Networks](https://mlsanity.com/paper/2201.01363/) (2022)
- [SIMD-size aware weight regularization for fast neural vocoding on CPU](https://mlsanity.com/paper/2211.00898/) (2022)
- [Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance](https://mlsanity.com/paper/2403.11100/) (2024)

## Abstract (arXiv)

Sparseness is a useful regularizer for learning in a wide range of applications, in particular in neural networks. This paper proposes a model targeted at classification tasks, where sparse activity and sparse connectivity are used to enhance classification capabilities. The tool for achieving this is a sparseness-enforcing projection operator which finds the closest vector with a pre-defined sparseness for any given vector. In the theoretical part of this paper, a comprehensive theory for such a projection is developed. In conclusion, it is shown that the projection is differentiable almost everywhere and can thus be implemented as a smooth neuronal transfer function. The entire model can hence be tuned end-to-end using gradient-based methods. Experiments on the MNIST database of handwritten digits show that classification performance can be boosted by sparse activity or sparse connectivity. With a combination of both, performance can be significantly better compared to classical non-sparse approaches.