# Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity

arXiv: https://arxiv.org/abs/2107.02306 · HTML: https://mlsanity.com/paper/2107.02306/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Compressing Deep Convolutional Networks using Vector Quantization](https://mlsanity.com/paper/1412.6115/) (2014)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Soft Threshold Weight Reparameterization for Learnable Sparsity](https://mlsanity.com/paper/2002.03231/) (2020)

## Built on by (most cited first, 14 of 14)

- [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Trainability Preserving Neural Pruning](https://mlsanity.com/paper/2207.12534/) (2022)
- [2SSP: A Two-Stage Framework for Structured Pruning of LLMs](https://mlsanity.com/paper/2501.17771/) (2025)
- [Structured Pruning is All You Need for Pruning CNNs at Initialization](https://mlsanity.com/paper/2203.02549/) (2022)
- [NIRVANA: Structured Pruning Reimagined for Large Language Model Compression](https://mlsanity.com/paper/2509.14230/) (2025)
- [One-shot Network Pruning at Initialization with Discriminative Image Patches](https://mlsanity.com/paper/2209.05683/) (2022)
- [Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations](https://mlsanity.com/paper/2305.16886/) (2023)
- [On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee](https://mlsanity.com/paper/2303.06815/) (2023)
- [CoNNect: Connectivity-Based Regularization for Structural Pruning](https://mlsanity.com/paper/2502.00744/) (2025)
- [Neural Network Panning: Screening the Optimal Sparse Network Before Training](https://mlsanity.com/paper/2209.13378/) (2022)
- [Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks](https://mlsanity.com/paper/2212.06144/) (2022)
- [Sparsest Models Elude Pruning: An Expos'e of Pruning's Current Capabilities](https://mlsanity.com/paper/2407.04075/) (2024)
- [Learning effective pruning at initialization from iterative pruning](https://mlsanity.com/paper/2408.14757/) (2024)

## Abstract (arXiv)

Neural network pruning is a fruitful area of research with surging interest in high sparsity regimes. Benchmarking in this domain heavily relies on faithful representation of the sparsity of subnetworks, which has been traditionally computed as the fraction of removed connections (direct sparsity). This definition, however, fails to recognize unpruned parameters that detached from input or output layers of underlying subnetworks, potentially underestimating actual effective sparsity: the fraction of inactivated connections. While this effect might be negligible for moderately pruned networks (up to 10-100 compression rates), we find that it plays an increasing role for thinner subnetworks, greatly distorting comparison between different pruning algorithms. For example, we show that effective compression of a randomly pruned LeNet-300-100 can be orders of magnitude larger than its direct counterpart, while no discrepancy is ever observed when using SynFlow for pruning [Tanaka et al., 2020]. In this work, we adopt the lens of effective sparsity to reevaluate several recent pruning algorithms on common benchmark architectures (e.g., LeNet-300-100, VGG-19, ResNet-18) and discover that their absolute and relative performance changes dramatically in this new and more appropriate framework. To aim for effective, rather than direct, sparsity, we develop a low-cost extension to most pruning algorithms. Further, equipped with effective sparsity as a reference frame, we partially reconfirm that random pruning with appropriate sparsity allocation across layers performs as well or better than more sophisticated algorithms for pruning at initialization [Su et al., 2020]. In response to this observation, using a simple analogy of pressure distribution in coupled cylinders from physics, we design novel layerwise sparsity quotas that outperform all existing baselines in the context of random pruning.