# Sparsely Activated Networks

arXiv: https://arxiv.org/abs/1907.06592 · HTML: https://mlsanity.com/paper/1907.06592/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2019.

## Built on by (most cited first, 5 of 5)

- [Deep Neural Network Initialization with Sparsity Inducing Activations](https://mlsanity.com/paper/2402.16184/) (2024)
- [The SkipSponge Attack: Sponge Weight Poisoning of Deep Neural Networks](https://mlsanity.com/paper/2402.06357/) (2024)
- [Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Bounds](https://mlsanity.com/paper/2505.15013/) (2025)
- [SPARLING: Learning Latent Representations with Extremely Sparse Activations](https://mlsanity.com/paper/2302.01976/) (2023)
- [Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation](https://mlsanity.com/paper/2509.15460/) (2025)

## Abstract (arXiv)

Previous literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features. However, this was done without considering the description length of the learned representations which is a direct and unbiased measure of the model complexity. In this paper, first we introduce the varphi metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined varphi. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using varphi have small description representation length and consist of interpretable kernels.