# Structured Pruning of Neural Networks with Budget-Aware Regularization

arXiv: https://arxiv.org/abs/1811.09332 · HTML: https://mlsanity.com/paper/1811.09332/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2018.

## Built on by (most cited first, 26 of 26)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [Efficient Neural Network Training via Forward and Backward Propagation Sparsification](https://mlsanity.com/paper/2111.05685/) (2021)
- [Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints](https://mlsanity.com/paper/2208.04425/) (2022)
- [ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations](https://mlsanity.com/paper/2102.07156/) (2021)
- [Towards Accurate and Compact Architectures via Neural Architecture Transformer](https://mlsanity.com/paper/2102.10301/) (2021)
- [Network Pruning using Adaptive Exemplar Filters](https://mlsanity.com/paper/2101.07985/) (2021)
- [Differentiable Network Pruning for Microcontrollers](https://mlsanity.com/paper/2110.08350/) (2021)
- [Differentiable Transportation Pruning](https://mlsanity.com/paper/2307.08483/) (2023)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [OptG: Optimizing Gradient-driven Criteria in Network Sparsity](https://mlsanity.com/paper/2201.12826/) (2022)
- [TinyM2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices](https://mlsanity.com/paper/2202.04303/) (2022)
- [Dynamic Kernel Selection for Improved Generalization and Memory Efficiency in Meta-learning](https://mlsanity.com/paper/2206.01690/) (2022)
- [Filter Pruning based on Information Capacity and Independence](https://mlsanity.com/paper/2303.03645/) (2023)
- [Structured Directional Pruning via Perturbation Orthogonal Projection](https://mlsanity.com/paper/2107.05328/) (2021)
- [Neural Network Pruning Through Constrained Reinforcement Learning](https://mlsanity.com/paper/2110.08558/) (2021)
- [LayerCollapse: Adaptive compression of neural networks](https://mlsanity.com/paper/2311.17943/) (2023)
- [Rescaling CNN through Learnable Repetition of Network Parameters](https://mlsanity.com/paper/2101.05650/) (2021)
- [KCP: Kernel Cluster Pruning for Dense Labeling Neural Networks](https://mlsanity.com/paper/2101.06686/) (2021)
- [Cogradient Descent for Dependable Learning](https://mlsanity.com/paper/2106.10617/) (2021)
- [Partial Binarization of Neural Networks for Budget-Aware Efficient Learning](https://mlsanity.com/paper/2211.06739/) (2022)
- [Training Lightweight Graph Convolutional Networks with Phase-field Models](https://mlsanity.com/paper/2212.09415/) (2022)
- [One-Shot Multi-Rate Pruning of Graph Convolutional Networks](https://mlsanity.com/paper/2312.17615/) (2023)
- [Designing Semi-Structured Pruning of Graph Convolutional Networks for Skeleton-based Recognition](https://mlsanity.com/paper/2412.11813/) (2024)
- [Exploring Vision Neural Network Pruning via Screening Methodology](https://mlsanity.com/paper/2502.07189/) (2025)

## Abstract (arXiv)

Pruning methods have shown to be effective at reducing the size of deep neural networks while keeping accuracy almost intact. Among the most effective methods are those that prune a network while training it with a sparsity prior loss and learnable dropout parameters. A shortcoming of these approaches however is that neither the size nor the inference speed of the pruned network can be controlled directly; yet this is a key feature for targeting deployment of CNNs on low-power hardware. To overcome this, we introduce a budgeted regularized pruning framework for deep CNNs. Our approach naturally fits into traditional neural network training as it consists of a learnable masking layer, a novel budget-aware objective function, and the use of knowledge distillation. We also provide insights on how to prune a residual network and how this can lead to new architectures. Experimental results reveal that CNNs pruned with our method are more accurate and less compute-hungry than state-of-the-art methods. Also, our approach is more effective at preventing accuracy collapse in case of severe pruning; this allows us to attain pruning factors up to 16x without significant accuracy drop.