# On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks

arXiv: https://arxiv.org/abs/2001.05050 · HTML: https://mlsanity.com/paper/2001.05050/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 13 of 13)

- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic](https://mlsanity.com/paper/2408.13656/) (2024)
- [HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers](https://mlsanity.com/paper/2302.09632/) (2023)
- [Pruning in the Face of Adversaries](https://mlsanity.com/paper/2108.08560/) (2021)
- [Zeroth-Order Topological Insights into Iterative Magnitude Pruning](https://mlsanity.com/paper/2206.06563/) (2022)
- [Towards Compute-Optimal Transfer Learning](https://mlsanity.com/paper/2304.13164/) (2023)
- [Quantifying lottery tickets under label noise: accuracy, calibration, and complexity](https://mlsanity.com/paper/2306.12190/) (2023)
- [Compressed Object Detection](https://mlsanity.com/paper/2102.02896/) (2021)
- [CRoP: Context-wise Robust Static Human-Sensing Personalization](https://mlsanity.com/paper/2409.17994/) (2024)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)
- [An End-to-End Network Pruning Pipeline with Sparsity Enforcement](https://mlsanity.com/paper/2312.01653/) (2023)
- [CELLM: An Efficient Communication in Large Language Models Training for Federated Learning](https://mlsanity.com/paper/2407.20557/) (2024)
- [As easy as PIE: understanding when pruning causes language models to disagree](https://mlsanity.com/paper/2503.21714/) (2025)

## Abstract (arXiv)

We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity sub-networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based unstructured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.