# N2NSkip: Learning Highly Sparse Networks using Neuron-to-Neuron Skip Connections

arXiv: https://arxiv.org/abs/2208.03662 · HTML: https://mlsanity.com/paper/2208.03662/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 7 Aug 2022.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [Learning heat diffusion graphs](https://mlsanity.com/paper/1611.01456/) (2016)
- [Deep Expander Networks: Efficient Deep Networks from Graph Theory](https://mlsanity.com/paper/1711.08757/) (2017)

## Built on by (most cited first, 9 of 9)

- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [One-shot Network Pruning at Initialization with Discriminative Image Patches](https://mlsanity.com/paper/2209.05683/) (2022)
- [WeightMom: Learning Sparse Networks using Iterative Momentum-based pruning](https://mlsanity.com/paper/2208.05970/) (2022)
- [Decay Pruning Method: Smooth Pruning With a Self-Rectifying Procedure](https://mlsanity.com/paper/2406.03879/) (2024)
- [Visual Heart Rate Estimation from RGB Facial Video using Spectral Reflectance](https://mlsanity.com/paper/2208.04947/) (2022)
- [Remote Photoplethysmography from Low Resolution videos: An end-to-end solution using Efficient ConvNets](https://mlsanity.com/paper/2208.06817/) (2022)
- [Memristive Computing for Efficient Inference on Resource Constrained Devices](https://mlsanity.com/paper/2208.10490/) (2022)
- [Minute ventilation measurement using Plethysmographic Imaging and lighting parameters](https://mlsanity.com/paper/2208.13319/) (2022)
- [EAPruning: Evolutionary Pruning for Vision Transformers and CNNs](https://mlsanity.com/paper/2210.00181/) (2022)

## Abstract (arXiv)

The over-parametrized nature of Deep Neural Networks leads to considerable hindrances during deployment on low-end devices with time and space constraints. Network pruning strategies that sparsify DNNs using iterative prune-train schemes are often computationally expensive. As a result, techniques that prune at initialization, prior to training, have become increasingly popular. In this work, we propose neuron-to-neuron skip connections, which act as sparse weighted skip connections, to enhance the overall connectivity of pruned DNNs. Following a preliminary pruning step, N2NSkip connections are randomly added between individual neurons/channels of the pruned network, while maintaining the overall sparsity of the network. We demonstrate that introducing N2NSkip connections in pruned networks enables significantly superior performance, especially at high sparsity levels, as compared to pruned networks without N2NSkip connections. Additionally, we present a heat diffusion-based connectivity analysis to quantitatively determine the connectivity of the pruned network with respect to the reference network. We evaluate the efficacy of our approach on two different preliminary pruning methods which prune at initialization, and consistently obtain superior performance by exploiting the enhanced connectivity resulting from N2NSkip connections.