# Zeroth-Order Topological Insights into Iterative Magnitude Pruning

arXiv: https://arxiv.org/abs/2206.06563 · HTML: https://mlsanity.com/paper/2206.06563/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Deep Learning with Topological Signatures](https://mlsanity.com/paper/1707.04041/) (2017)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology](https://mlsanity.com/paper/1812.09764/) (2018)
- [Geometry Score: A Method For Comparing Generative Adversarial Networks](https://mlsanity.com/paper/1802.02664/) (2018)
- [On Characterizing the Capacity of Neural Networks using Algebraic Topology](https://mlsanity.com/paper/1802.04443/) (2018)
- [A Gradient Flow Framework For Analyzing Network Pruning](https://mlsanity.com/paper/2009.11839/) (2020)
- [Robust Pruning at Initialization](https://mlsanity.com/paper/2002.08797/) (2020)
- [Topological Approaches to Deep Learning](https://mlsanity.com/paper/1811.01122/) (2018)
- [Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning](https://mlsanity.com/paper/2007.08243/) (2020)
- [Topological Uncertainty: Monitoring trained neural networks through persistence of activation graphs](https://mlsanity.com/paper/2105.04404/) (2021)
- [Topological Data Analysis of Decision Boundaries with Application to Model Selection](https://mlsanity.com/paper/1805.09949/) (2018)
- [A Unified Paths Perspective for Pruning at Initialization](https://mlsanity.com/paper/2101.10552/) (2021)

## Built on by (most cited first, 2 of 2)

- [Addressing caveats of neural persistence with deep graph persistence](https://mlsanity.com/paper/2307.10865/) (2023)
- [Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis](https://mlsanity.com/paper/2308.03128/) (2023)

## Abstract (arXiv)

Modern-day neural networks are famously large, yet also highly redundant and compressible; there exist numerous pruning strategies in the deep learning literature that yield over 90% sparser sub-networks of fully-trained, dense architectures while still maintaining their original accuracies. Amongst these many methods though -- thanks to its conceptual simplicity, ease of implementation, and efficacy -- Iterative Magnitude Pruning (IMP) dominates in practice and is the de facto baseline to beat in the pruning community. However, theoretical explanations as to why a simplistic method such as IMP works at all are few and limited. In this work, we leverage the notion of persistent homology to gain insights into the workings of IMP and show that it inherently encourages retention of those weights which preserve topological information in a trained network. Subsequently, we also provide bounds on how much different networks can be pruned while perfectly preserving their zeroth order topological features, and present a modified version of IMP to do the same.