# Training Behavior of Sparse Neural Network Topologies

arXiv: https://arxiv.org/abs/1810.00299 · HTML: https://mlsanity.com/paper/1810.00299/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2018.

## Built on by (most cited first, 6 of 6)

- [Truly Sparse Neural Networks at Scale](https://mlsanity.com/paper/2102.01732/) (2021)
- [The Effect of Model Compression on Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2201.01709/) (2022)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)
- [Testing RadiX-Nets: Advances in Viable Sparse Topologies](https://mlsanity.com/paper/2311.03609/) (2023)
- [Solving hybrid machine learning tasks by traversing weight space geodesics](https://mlsanity.com/paper/2106.02793/) (2021)
- [What to Prune and What Not to Prune at Initialization](https://mlsanity.com/paper/2209.02201/) (2022)

## Abstract (arXiv)

Improvements in the performance of deep neural networks have often come through the design of larger and more complex networks. As a result, fast memory is a significant limiting factor in our ability to improve network performance. One approach to overcoming this limit is the design of sparse neural networks, which can be both very large and efficiently trained. In this paper we experiment training on sparse neural network topologies. We test pruning-based topologies, which are derived from an initially dense network whose connections are pruned, as well as RadiX-Nets, a class of network topologies with proven connectivity and sparsity properties. Results show that sparse networks obtain accuracies comparable to dense networks, but extreme levels of sparsity cause instability in training, which merits further study.