# On improving deep learning generalization with adaptive sparse connectivity

arXiv: https://arxiv.org/abs/1906.11626 · HTML: https://mlsanity.com/paper/1906.11626/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2019.

## Built on by (most cited first, 7 of 7)

- [Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.06548/) (2023)
- [Dynamic Sparse Training of Diagonally Sparse Networks](https://mlsanity.com/paper/2506.11449/) (2025)
- [Visual Attention Emerges from Recurrent Sparse Reconstruction](https://mlsanity.com/paper/2204.10962/) (2022)
- [Sparse deep neural networks for modeling aluminum electrolysis dynamics](https://mlsanity.com/paper/2209.05832/) (2022)
- [Estimating the Generalization in Deep Neural Networks via Sparsity](https://mlsanity.com/paper/2104.00851/) (2021)
- [Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2508.06871/) (2025)
- [SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks](https://mlsanity.com/paper/2510.14812/) (2025)

## Abstract (arXiv)

Large neural networks are very successful in various tasks. However, with limited data, the generalization capabilities of deep neural networks are also very limited. In this paper, we empirically start showing that intrinsically sparse neural networks with adaptive sparse connectivity, which by design have a strict parameter budget during the training phase, have better generalization capabilities than their fully-connected counterparts. Besides this, we propose a new technique to train these sparse models by combining the Sparse Evolutionary Training (SET) procedure with neurons pruning. Operated on MultiLayer Perceptron (MLP) and tested on 15 datasets, our proposed technique zeros out around 50% of the hidden neurons during training, while having a linear number of parameters to optimize with respect to the number of neurons. The results show a competitive classification and generalization performance.