mlsanityData to 7 Oct 2026

Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask

arXiv 2209.07617 ↗First posted 15 Sep 2022cs.LG cs.AI cs.ARcited by 10

Not on the map. Why?

Abstract (arXiv)

Sparsity has become one of the promising methods to compress and accelerate Deep Neural Networks (DNNs). Among different categories of sparsity, structured sparsity has gained more attention due to its efficient execution on modern accelerators. Particularly, N:M sparsity is attractive because there are already hardware accelerator architectures that can leverage certain forms of N:M structured sparsity to yield higher compute-efficiency. In this work, we focus on N:M sparsity and extensively study and evaluate various training recipes for N:M sparsity in terms of the trade-off between model accuracy and compute cost (FLOPs). Building upon this study, we propose two new decay-based pruning methods, namely "pruning mask decay" and "sparse structure decay". Our evaluations indicate that these proposed methods consistently deliver state-of-the-art (SOTA) model accuracy, comparable to unstructured sparsity, on a Transformer-based model for a translation task. The increase in the accuracy of the sparse model using the new training recipes comes at the cost of marginal increase in the total training compute (FLOPs).

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

51 of 59 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
38 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  2. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  3. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  4. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  5. 2018Rethinking the Value of Network Pruningcited by 595
  6. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  7. 2019Importance Estimation for Neural Network Pruningcited by 531
  8. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2019The State of Sparsity in Deep Neural Networkscited by 471
  11. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  12. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  13. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  14. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  15. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  16. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  17. 2018Efficient Neural Audio Synthesiscited by 218
  18. 2021Accelerating Sparse Deep Neural Networkscited by 213
  19. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  20. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  21. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  22. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  23. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  24. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  25. 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
  26. 2019Fast Sparse ConvNetscited by 86
  27. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  28. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  29. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  30. 2019Discovering Neural Wiringscited by 56
  31. 2017Block-Sparse Recurrent Neural Networkscited by 38
  32. 2019Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?cited by 31
  33. 2018Balanced Sparsity for Efficient DNN Inference on GPUcited by 27
  34. 2018Accelerator-Aware Pruning for Convolutional Neural Networkscited by 22
  35. 2020Filter Sketch for Network Pruningcited by 20
  36. 2020PCNN: Pattern-based Fine-Grained Regular Pruning towards Optimizing CNN Acceleratorscited by 9
  37. 2020Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Acceleratorscited by 7
  38. 2021Extending Sparse Tensor Accelerators to Support Multiple Compression Formatscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 1.

10 papers.

  1. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  2. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 25
  3. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 18
  4. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  5. 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 10
  6. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  7. 2025Pruning for Sparse Diffusion Models based on Gradient Flowcited by 5
  8. 2023Just CHOP: Embarrassingly Simple LLM Compressioncited by 4
  9. 2025PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Modelscited by 1
  10. 2026GroupMask: Layer-Adaptive Group-wise Sparsity for Semi-Structured LLM Pruningcited by 0