# A Unified Paths Perspective for Pruning at Initialization

arXiv: https://arxiv.org/abs/2101.10552 · HTML: https://mlsanity.com/paper/2101.10552/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 12 of 12)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [A Signal Propagation Perspective for Pruning Neural Networks at Initialization](https://mlsanity.com/paper/1906.06307/) (2019)
- [How Important Is a Neuron?](https://mlsanity.com/paper/1805.12233/) (2018)
- [Finding trainable sparse networks through Neural Tangent Transfer](https://mlsanity.com/paper/2006.08228/) (2020)
- [G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space](https://mlsanity.com/paper/1802.03713/) (2018)

## Built on by (most cited first, 14 of 14)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [What can linearized neural networks actually say about generalization?](https://mlsanity.com/paper/2106.06770/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning](https://mlsanity.com/paper/2406.01820/) (2024)
- [Neural Lineage](https://mlsanity.com/paper/2406.11129/) (2024)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [Path-conditioned training: a principled way to rescale ReLU neural networks](https://mlsanity.com/paper/2602.19799/) (2026)
- [Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think](https://mlsanity.com/paper/2211.17180/) (2022)
- [Intrinsic training dynamics of deep neural networks](https://mlsanity.com/paper/2508.07370/) (2025)
- [Zeroth-Order Topological Insights into Iterative Magnitude Pruning](https://mlsanity.com/paper/2206.06563/) (2022)
- [Pruning at Initialization -- A Sketching Perspective](https://mlsanity.com/paper/2305.17559/) (2023)
- [The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis](https://mlsanity.com/paper/2510.17515/) (2025)
- [Gradual Capacity Growth for Sparse Network Discovery](https://mlsanity.com/paper/2509.25665/) (2025)

## Abstract (arXiv)

A number of recent approaches have been proposed for pruning neural network parameters at initialization with the goal of reducing the size and computational burden of models while minimally affecting their training dynamics and generalization performance. While each of these approaches have some amount of well-founded motivation, a rigorous analysis of the effect of these pruning methods on network training dynamics and their formal relationship to each other has thus far received little attention. Leveraging recent theoretical approximations provided by the Neural Tangent Kernel, we unify a number of popular approaches for pruning at initialization under a single path-centric framework. We introduce the Path Kernel as the data-independent factor in a decomposition of the Neural Tangent Kernel and show the global structure of the Path Kernel can be computed efficiently. This Path Kernel decomposition separates the architectural effects from the data-dependent effects within the Neural Tangent Kernel, providing a means to predict the convergence dynamics of a network from its architecture alone. We analyze the use of this structure in approximating training and generalization performance of networks in the absence of data across a number of initialization pruning approaches. Observing the relationship between input data and paths and the relationship between the Path Kernel and its natural norm, we additionally propose two augmentations of the SynFlow algorithm for pruning at initialization.