# Directional Pruning of Deep Neural Networks

arXiv: https://arxiv.org/abs/2006.09358 · HTML: https://mlsanity.com/paper/2006.09358/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution](https://mlsanity.com/paper/2204.08397/) (2022)
- [Anchor-based Plain Net for Mobile Image Super-Resolution](https://mlsanity.com/paper/2105.09750/) (2021)
- [Memorize, Factorize, or be Na"ive: Learning Optimal Feature Interaction Methods for CTR Prediction](https://mlsanity.com/paper/2108.01265/) (2021)
- [A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance](https://mlsanity.com/paper/2308.13504/) (2023)
- [Ex uno plures: Splitting One Model into an Ensemble of Subnetworks](https://mlsanity.com/paper/2106.04767/) (2021)
- [Sparse Personalized Federated Learning](https://mlsanity.com/paper/2107.05330/) (2021)
- [Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance](https://mlsanity.com/paper/2301.13376/) (2023)
- [ONNXPruner: ONNX-Based General Model Pruning Adapter](https://mlsanity.com/paper/2404.08016/) (2024)
- [Efficient Inference via Universal LSH Kernel](https://mlsanity.com/paper/2106.11426/) (2021)
- [A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks](https://mlsanity.com/paper/2201.08652/) (2022)
- [EsaCL: Efficient Continual Learning of Sparse Models](https://mlsanity.com/paper/2401.05667/) (2024)
- [Structured Directional Pruning via Perturbation Orthogonal Projection](https://mlsanity.com/paper/2107.05328/) (2021)
- [State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks](https://mlsanity.com/paper/2209.09389/) (2022)
- [Sebica: Lightweight Spatial and Efficient Bidirectional Channel Attention Super Resolution Network](https://mlsanity.com/paper/2410.20546/) (2024)

## Abstract (arXiv)

In the light of the fact that the stochastic gradient descent (SGD) often finds a flat minimum valley in the training loss, we propose a novel directional pruning method which searches for a sparse minimizer in or close to that flat region. The proposed pruning method does not require retraining or the expert knowledge on the sparsity level. To overcome the computational formidability of estimating the flat directions, we propose to use a carefully tuned ℓ1 proximal gradient algorithm which can provably achieve the directional pruning with a small learning rate after sufficient training. The empirical results demonstrate the promising results of our solution in highly sparse regime (92% sparsity) among many existing pruning methods on the ResNet50 with the ImageNet, while using only a slightly higher wall time and memory footprint than the SGD. Using the VGG16 and the wide ResNet 28x10 on the CIFAR-10 and CIFAR-100, we demonstrate that our solution reaches the same minima valley as the SGD, and the minima found by our solution and the SGD do not deviate in directions that impact the training loss. The code that reproduces the results of this paper is available at https://github.com/donlan2710/gRDA-Optimizer/tree/master/directionalpruning.