# DRIVE: Dual Gradient-Based Rapid Iterative Pruning

arXiv: https://arxiv.org/abs/2404.03687 · HTML: https://mlsanity.com/paper/2404.03687/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [A Signal Propagation Perspective for Pruning Neural Networks at Initialization](https://mlsanity.com/paper/1906.06307/) (2019)
- [Fast Sparse ConvNets](https://mlsanity.com/paper/1911.09723/) (2019)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [Progressive Skeletonization: Trimming more fat from a network at initialization](https://mlsanity.com/paper/2006.09081/) (2020)
- [PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration](https://mlsanity.com/paper/1901.09290/) (2019)
- [Stabilizing the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1903.01611/) (2019)
- [SparseDNN: Fast Sparse Deep Learning Inference on CPUs](https://mlsanity.com/paper/2101.07948/) (2021)
- [Pre-Defined Sparse Neural Networks with Hardware Acceleration](https://mlsanity.com/paper/1812.01164/) (2018)

## Built on by (most cited first, 3 of 3)

- [RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models](https://mlsanity.com/paper/2510.01240/) (2025)
- [TinyEcoWeedNet: Edge Efficient Real-Time Aerial Agricultural Weed Detection](https://mlsanity.com/paper/2509.18193/) (2025)
- [A Recovery Guarantee for Sparse Neural Networks](https://mlsanity.com/paper/2509.20323/) (2025)

## Abstract (arXiv)

Modern deep neural networks (DNNs) consist of millions of parameters, necessitating high-performance computing during training and inference. Pruning is one solution that significantly reduces the space and time complexities of DNNs. Traditional pruning methods that are applied post-training focus on streamlining inference, but there are recent efforts to leverage sparsity early on by pruning before training. Pruning methods, such as iterative magnitude-based pruning (IMP) achieve up to a 90% parameter reduction while retaining accuracy comparable to the original model. However, this leads to impractical runtime as it relies on multiple train-prune-reset cycles to identify and eliminate redundant parameters. In contrast, training agnostic early pruning methods, such as SNIP and SynFlow offer fast pruning but fall short of the accuracy achieved by IMP at high sparsities. To bridge this gap, we present Dual Gradient-Based Rapid Iterative Pruning (DRIVE), which leverages dense training for initial epochs to counteract the randomness inherent at the initialization. Subsequently, it employs a unique dual gradient-based metric for parameter ranking. It has been experimentally demonstrated for VGG and ResNet architectures on CIFAR-10/100 and Tiny ImageNet, and ResNet on ImageNet that DRIVE consistently has superior performance over other training-agnostic early pruning methods in accuracy. Notably, DRIVE is 43× to 869× faster than IMP for pruning.