# Sparse Super-Regular Networks

arXiv: https://arxiv.org/abs/2201.01363 · HTML: https://mlsanity.com/paper/2201.01363/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 4 Jan 2022.

## Builds on (most cited first, 8 of 8)

- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Sparse DNNs with Improved Adversarial Robustness](https://mlsanity.com/paper/1810.09619/) (2018)
- [Learning Sparse Neural Networks via Sensitivity-Driven Regularization](https://mlsanity.com/paper/1810.11764/) (2018)
- [Deep Expander Networks: Efficient Deep Networks from Graph Theory](https://mlsanity.com/paper/1711.08757/) (2017)
- [Sparsifying Neural Network Connections for Face Recognition](https://mlsanity.com/paper/1512.01891/) (2015)
- [Sparse Activity and Sparse Connectivity in Supervised Learning](https://mlsanity.com/paper/1603.08367/) (2016)

## Built on by (most cited first, 1 of 1)

- [Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance](https://mlsanity.com/paper/2403.11100/) (2024)

## Abstract (arXiv)

It has been argued by Thom and Palm that sparsely-connected neural networks (SCNs) show improved performance over fully-connected networks (FCNs). Super-regular networks (SRNs) are neural networks composed of a set of stacked sparse layers of (epsilon, delta)-super-regular pairs, and randomly permuted node order. Using the Blow-up Lemma, we prove that as a result of the individual super-regularity of each pair of layers, SRNs guarantee a number of properties that make them suitable replacements for FCNs for many tasks. These guarantees include edge uniformity across all large-enough subsets, minimum node in- and out-degree, input-output sensitivity, and the ability to embed pre-trained constructs. Indeed, SRNs have the capacity to act like FCNs, and eliminate the need for costly regularization schemes like Dropout. We show that SRNs perform similarly to X-Nets via readily reproducible experiments, and offer far greater guarantees and control over network structure.