# A Gradient Flow Framework For Analyzing Network Pruning

arXiv: https://arxiv.org/abs/2009.11839 · HTML: https://mlsanity.com/paper/2009.11839/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2020.

## Built on by (most cited first, 30 of 36)

- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks](https://mlsanity.com/paper/2311.12786/) (2023)
- [Rare Gems: Finding Lottery Tickets at Initialization](https://mlsanity.com/paper/2202.12002/) (2022)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers](https://mlsanity.com/paper/2302.09632/) (2023)
- [Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization](https://mlsanity.com/paper/2102.01670/) (2021)
- [Winning the Lottery Ahead of Time: Efficient Early Network Pruning](https://mlsanity.com/paper/2206.10451/) (2022)
- [Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective](https://mlsanity.com/paper/2310.11451/) (2023)
- [Trainability Preserving Neural Pruning](https://mlsanity.com/paper/2207.12534/) (2022)
- [Sparse maximal update parameterization: A holistic approach to sparse training dynamics](https://mlsanity.com/paper/2405.15743/) (2024)
- [No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models](https://mlsanity.com/paper/2202.02664/) (2022)
- [DiSparse: Disentangled Sparsification for Multitask Model Compression](https://mlsanity.com/paper/2206.04662/) (2022)
- [Towards Structured Dynamic Sparse Pre-Training of BERT](https://mlsanity.com/paper/2108.06277/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics](https://mlsanity.com/paper/2411.16139/) (2024)
- [Structurally Prune Anything: Any Architecture, Any Framework, Any Time](https://mlsanity.com/paper/2403.18955/) (2024)
- [Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood](https://mlsanity.com/paper/2402.15978/) (2024)
- [The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains](https://mlsanity.com/paper/2205.11416/) (2022)
- [PAC-Net: A Model Pruning Approach to Inductive Transfer Learning](https://mlsanity.com/paper/2206.05703/) (2022)
- [Dynamic Probabilistic Pruning: A general framework for hardware-constrained pruning at different granularities](https://mlsanity.com/paper/2105.12686/) (2021)
- [Pruning for Sparse Diffusion Models based on Gradient Flow](https://mlsanity.com/paper/2501.09464/) (2025)
- [Neural Network Pruning by Gradient Descent](https://mlsanity.com/paper/2311.12526/) (2023)
- [F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics](https://mlsanity.com/paper/2411.11912/) (2024)
- [Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning](https://mlsanity.com/paper/2503.21419/) (2025)
- [Zeroth-Order Topological Insights into Iterative Magnitude Pruning](https://mlsanity.com/paper/2206.06563/) (2022)
- [One-shot Network Pruning at Initialization with Discriminative Image Patches](https://mlsanity.com/paper/2209.05683/) (2022)
- [Language-Aware Multilingual Machine Translation with Self-Supervised Learning](https://mlsanity.com/paper/2302.05008/) (2023)
- [The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis](https://mlsanity.com/paper/2510.17515/) (2025)

## Abstract (arXiv)

Recent network pruning methods focus on pruning models early-on in training. To estimate the impact of removing a parameter, these methods use importance measures that were originally designed to prune trained models. Despite lacking justification for their use early-on in training, such measures result in surprisingly low accuracy loss. To better explain this behavior, we develop a general framework that uses gradient flow to unify state-of-the-art importance measures through the norm of model parameters. We use this framework to determine the relationship between pruning measures and evolution of model parameters, establishing several results related to pruning models early-on in training: (i) magnitude-based pruning removes parameters that contribute least to reduction in loss, resulting in models that converge faster than magnitude-agnostic methods; (ii) loss-preservation based pruning preserves first-order model evolution dynamics and is therefore appropriate for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gradient norm via pruning can produce poorly performing models. We validate our claims on several VGG-13, MobileNet-V1, and ResNet-56 models trained on CIFAR-10/CIFAR-100. Code available at https://github.com/EkdeepSLubana/flowandprune.