# DARB: A Density-Aware Regular-Block Pruning for Deep Neural Networks

arXiv: https://arxiv.org/abs/1911.08020 · HTML: https://mlsanity.com/paper/1911.08020/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2019.

## Built on by (most cited first, 5 of 5)

- [Quantum Neural Network Compression](https://mlsanity.com/paper/2207.01578/) (2022)
- [A Secure and Efficient Federated Learning Framework for NLP](https://mlsanity.com/paper/2201.11934/) (2022)
- [Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization](https://mlsanity.com/paper/2101.04073/) (2021)
- [Load-balanced Gather-scatter Patterns for Sparse Deep Neural Networks](https://mlsanity.com/paper/2112.10898/) (2021)
- [A One-Shot Reparameterization Method for Reducing the Loss of Tile Pruning on DNNs](https://mlsanity.com/paper/2207.14545/) (2022)

## Abstract (arXiv)

The rapidly growing parameter volume of deep neural networks (DNNs) hinders the artificial intelligence applications on resource constrained devices, such as mobile and wearable devices. Neural network pruning, as one of the mainstream model compression techniques, is under extensive study to reduce the number of parameters and computations. In contrast to irregular pruning that incurs high index storage and decoding overhead, structured pruning techniques have been proposed as the promising solutions. However, prior studies on structured pruning tackle the problem mainly from the perspective of facilitating hardware implementation, without analyzing the characteristics of sparse neural networks. The neglect on the study of sparse neural networks causes inefficient trade-off between regularity and pruning ratio. Consequently, the potential of structurally pruning neural networks is not sufficiently mined. In this work, we examine the structural characteristics of the irregularly pruned weight matrices, such as the diverse redundancy of different rows, the sensitivity of different rows to pruning, and the positional characteristics of retained weights. By leveraging the gained insights as a guidance, we first propose the novel block-max weight masking (BMWM) method, which can effectively retain the salient weights while imposing high regularity to the weight matrix. As a further optimization, we propose a density-adaptive regular-block (DARB) pruning that outperforms prior structured pruning work with high pruning ratio and decoding efficiency. Our experimental results show that DARB can achieve 13× to 25× pruning ratio, which are 2.8× to 4.3× improvements than the state-of-the-art counterparts on multiple neural network models and tasks. Moreover, DARB can achieve 14.3× decoding efficiency than block pruning with higher pruning ratio.