# Data-Driven Sparse Structure Selection for Deep Neural Networks

arXiv: https://arxiv.org/abs/1707.01213 · HTML: https://mlsanity.com/paper/1707.01213/

## Key facts

- Cited by 181 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2017.

## Built on by (most cited first, 30 of 181)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Differentiable Subset Pruning of Transformer Heads](https://mlsanity.com/paper/2108.04657/) (2021)
- [Hessian-Aware Pruning and Optimal Neural Implant](https://mlsanity.com/paper/2101.08940/) (2021)
- [Learning to Weight Samples for Dynamic Early-exiting Networks](https://mlsanity.com/paper/2209.08310/) (2022)
- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [Dynamic Resolution Network](https://mlsanity.com/paper/2106.02898/) (2021)
- [STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://mlsanity.com/paper/2409.06211/) (2024)
- [DECORE: Deep Compression with Reinforcement Learning](https://mlsanity.com/paper/2106.06091/) (2021)
- [Optimal ablation for interpretability](https://mlsanity.com/paper/2409.09951/) (2024)
- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models](https://mlsanity.com/paper/2410.11988/) (2024)
- [BCNet: Searching for Network Width with Bilaterally Coupled Network](https://mlsanity.com/paper/2105.10533/) (2021)
- [On the Effect of Pruning on Adversarial Robustness](https://mlsanity.com/paper/2108.04890/) (2021)
- [Carrying out CNN Channel Pruning in a White Box](https://mlsanity.com/paper/2104.11883/) (2021)
- [Pruning for Robust Concept Erasing in Diffusion Models](https://mlsanity.com/paper/2405.16534/) (2024)

## Abstract (arXiv)

Deep convolutional neural networks have liberated its extraordinary power on various tasks. However, it is still very challenging to deploy state-of-the-art models into real-world applications due to their high computational complexity. How can we design a compact and effective network without massive experiments and expert knowledge? In this paper, we propose a simple and effective framework to learn and prune deep models in an end-to-end manner. In our framework, a new type of parameter -- scaling factor is first introduced to scale the outputs of specific structures, such as neurons, groups or residual blocks. Then we add sparsity regularizations on these factors, and solve this optimization problem by a modified stochastic Accelerated Proximal Gradient (APG) method. By forcing some of the factors to zero, we can safely remove the corresponding structures, thus prune the unimportant parts of a CNN. Comparing with other structure selection methods that may need thousands of trials or iterative fine-tuning, our method is trained fully end-to-end in one training pass without bells and whistles. We evaluate our method, Sparse Structure Selection with several state-of-the-art CNNs, and demonstrate very promising results with adaptive depth and width selection.