# Joint Multi-Dimension Pruning via Numerical Gradient Update

arXiv: https://arxiv.org/abs/2005.08931 · HTML: https://mlsanity.com/paper/2005.08931/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 7 of 7)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [Carrying out CNN Channel Pruning in a White Box](https://mlsanity.com/paper/2104.11883/) (2021)
- [Practical Network Acceleration with Tiny Sets](https://mlsanity.com/paper/2202.07861/) (2022)
- [Practical Network Acceleration with Tiny Sets: Hypothesis, Theory, and Algorithm](https://mlsanity.com/paper/2303.00972/) (2023)
- [Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler](https://mlsanity.com/paper/2307.00198/) (2023)
- [AutoSculpt: A Pattern-based Model Auto-pruning Framework Using Reinforcement Learning and Graph Learning](https://mlsanity.com/paper/2412.18091/) (2024)

## Abstract (arXiv)

We present joint multi-dimension pruning (abbreviated as JointPruning), an effective method of pruning a network on three crucial aspects: spatial, depth and channel simultaneously. To tackle these three naturally different dimensions, we proposed a general framework by defining pruning as seeking the best pruning vector (i.e., the numerical value of layer-wise channel number, spacial size, depth) and construct a unique mapping from the pruning vector to the pruned network structures. Then we optimize the pruning vector with gradient update and model joint pruning as a numerical gradient optimization process. To overcome the challenge that there is no explicit function between the loss and the pruning vectors, we proposed self-adapted stochastic gradient estimation to construct a gradient path through network loss to pruning vectors and enable efficient gradient update. We show that the joint strategy discovers a better status than previous studies that focused on individual dimensions solely, as our method is optimized collaboratively across the three dimensions in a single end-to-end training and it is more efficient than the previous exhaustive methods. Extensive experiments on large-scale ImageNet dataset across a variety of network architectures MobileNet V1&V2&V3 and ResNet demonstrate the effectiveness of our proposed method. For instance, we achieve significant margins of 2.5% and 2.6% improvement over the state-of-the-art approach on the already compact MobileNet V1&V2 under an extremely large compression ratio.