Data to 14 Oct 2026

Learning Structured Sparsity in Deep Neural Networks

arXiv 1608.03665 ↗First posted 12 Aug 2016cs.NE cs.LG stat.MLcited by 541

Not on the map. Why?

Abstract (arXiv)

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

Key facts. Cited by 541 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Aug 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 143; 2022: 106; 2023: 97; 2024: 93; 2025: 67; 2026: 35.

541 papers.

Sort:

By area:

2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
2023A Survey on Model Compression for Large Language Modelscited by 269
2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
2021NeRV: Neural Representations for Videoscited by 256
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
2021Accelerating Sparse Deep Neural Networkscited by 223
2021R-Drop: Regularized Dropout for Neural Networkscited by 202
2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 175
2021Patch Slimming for Efficient Vision Transformerscited by 133

38 more

2021Group Fisher Pruning for Practical Network Compressioncited by 131
2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 130
2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 129
2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
2021Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approachcited by 110
2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
2021Vision Transformer Pruningcited by 79
2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 78
2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
2022Compressing Volumetric Radiance Fields to 1 MBcited by 68
2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 67
2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 58
2021Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19 Diagnosis at the Edgecited by 56
2021CDFI: Compression-Driven Network Design for Frame Interpolationcited by 56
2022Advancing Model Pruning via Bi-level Optimizationcited by 56
2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 55
2021Anycost GANs for Interactive Image Synthesis and Editingcited by 54
2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
2023EVC: Towards Real-Time Neural Image Compression with Mask Decaycited by 49
2023Intriguing Properties of Quantization at Scalecited by 49
2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
2021Differentiable Subset Pruning of Transformer Headscited by 47
2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
2021Manifold Regularized Dynamic Network Pruningcited by 43
2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 43
2021Accelerating Large Scale Real-Time GNN Inference using Channel Pruningcited by 41
2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40

50 most cited of 541 All 541 (JSON)