# Layer Sparsity in Neural Networks

arXiv: https://arxiv.org/abs/2006.15604 · HTML: https://mlsanity.com/paper/2006.15604/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 6 of 6)

- [Activation Functions in Artificial Neural Networks: A Systematic Overview](https://mlsanity.com/paper/2101.09957/) (2021)
- [Statistical guarantees for sparse deep learning](https://mlsanity.com/paper/2212.05427/) (2022)
- [Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks](https://mlsanity.com/paper/2106.02260/) (2021)
- [How many samples are needed to train a deep neural network?](https://mlsanity.com/paper/2405.16696/) (2024)
- [Regularization can make diffusion models more efficient](https://mlsanity.com/paper/2502.09151/) (2025)
- [Improving the Deployment of Recycling Classification through Efficient Hyper-Parameter Analysis](https://mlsanity.com/paper/2110.11043/) (2021)

## Abstract (arXiv)

Sparsity has become popular in machine learning, because it can save computational resources, facilitate interpretations, and prevent overfitting. In this paper, we discuss sparsity in the framework of neural networks. In particular, we formulate a new notion of sparsity that concerns the networks' layers and, therefore, aligns particularly well with the current trend toward deep networks. We call this notion layer sparsity. We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks.