# ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations

arXiv: https://arxiv.org/abs/2102.07156 · HTML: https://mlsanity.com/paper/2102.07156/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 14 Feb 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Compressing Deep Convolutional Networks using Vector Quantization](https://mlsanity.com/paper/1412.6115/) (2014)
- [MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks](https://mlsanity.com/paper/1711.06798/) (2017)
- [Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure](https://mlsanity.com/paper/1904.03837/) (2019)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures](https://mlsanity.com/paper/1908.09979/) (2019)
- [Compression-aware Training of Deep Networks](https://mlsanity.com/paper/1711.02638/) (2017)
- [Structured Pruning of Neural Networks with Budget-Aware Regularization](https://mlsanity.com/paper/1811.09332/) (2018)

## Built on by (most cited first, 21 of 21)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [Layer Folding: Neural Network Depth Reduction using Activation Linearization](https://mlsanity.com/paper/2106.09309/) (2021)
- [GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer](https://mlsanity.com/paper/2301.05345/) (2023)
- [Differentiable Transportation Pruning](https://mlsanity.com/paper/2307.08483/) (2023)
- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)](https://mlsanity.com/paper/2110.00684/) (2021)
- [Dynamic Kernel Selection for Improved Generalization and Memory Efficiency in Meta-learning](https://mlsanity.com/paper/2206.01690/) (2022)
- [Filter Pruning based on Information Capacity and Independence](https://mlsanity.com/paper/2303.03645/) (2023)
- [Surgical Feature-Space Decomposition of LLMs: Why, When and How?](https://mlsanity.com/paper/2405.13039/) (2024)
- [B-FPGM: Lightweight Face Detection via Bayesian-Optimized Soft FPGM Pruning](https://mlsanity.com/paper/2501.16917/) (2025)
- [PRANC: Pseudo RAndom Networks for Compacting deep models](https://mlsanity.com/paper/2206.08464/) (2022)
- [Dynamic Channel Selection in Self-Supervised Learning](https://mlsanity.com/paper/2207.12065/) (2022)
- [On Designing Light-Weight Object Trackers through Network Pruning: Use CNNs or Transformers?](https://mlsanity.com/paper/2211.13769/) (2022)
- [Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programming](https://mlsanity.com/paper/2301.12187/) (2023)
- [Partial Binarization of Neural Networks for Budget-Aware Efficient Learning](https://mlsanity.com/paper/2211.06739/) (2022)
- [Unifying Synergies between Self-supervised Learning and Dynamic Computation](https://mlsanity.com/paper/2301.09164/) (2023)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)

## Abstract (arXiv)

Structured pruning methods are among the effective strategies for extracting small resource-efficient convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network. Our choice of continuous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible and expected as final outcomes. Our approach's flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.