mlsanityData to 14 Oct 2026

Learning Best Combination for Efficient N:M Sparsity

arXiv 2206.06662 ↗First posted 14 Jun 2022cs.LG cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

By forcing at most N out of M consecutive weights to be non-zero, the recent N:M network sparsity has received increasing attention for its two attractive advantages: 1) Promising performance at a high sparsity. 2) Significant speedups on NVIDIA A100 GPUs. Recent studies require an expensive pre-training phase or a heavy dense-gradient computation. In this paper, we show that the N:M learning can be naturally characterized as a combinatorial problem which searches for the best combination candidate within a finite collection. Motivated by this characteristic, we solve N:M sparsity in an efficient divide-and-conquer manner. First, we divide the weight vector into CMN combination subsets of a fixed size N. Then, we conquer the combinatorial problem by assigning each combination a learnable score that is jointly optimized with its associate weights. We prove that the introduced scoring mechanism can well model the relative importance between combination subsets. And by gradually removing low-scored subsets, N:M fine-grained sparsity can be efficiently optimized during the normal training phase. Comprehensive experiments demonstrate that our learning best combination (LBC) performs consistently better than off-the-shelf N:M sparsity methods across various networks. Our project is released at urlhttps://github.com/zyxxmu/LBC.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

34 of 56 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2017Mask R-CNNcited by 7,277
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2015Fast R-CNNcited by 2,933
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  12. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
22 more
  1. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  2. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  4. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  5. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  6. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 292
  7. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  8. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  9. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  10. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  11. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  12. 2020Channel Pruning via Automatic Structure Searchcited by 85
  13. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  14. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  15. 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
  16. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  17. 2020DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocationcited by 47
  18. 2019Approximated Oracle Filter Pruning for Destructive CNN Width Optimizationcited by 35
  19. 2020SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inferencecited by 30
  20. 2021Carrying out CNN Channel Pruning in a White Boxcited by 22
  21. 2018Unsupervised learning of foreground object detectioncited by 15
  22. 2022Minimum Variance Unbiased N:M Sparsity for the Neural Gradientscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 13; 2025: 10; 2026: 1.

32 papers.

Sort:

By area:

  1. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  2. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  3. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  4. 2024MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Maskscited by 41
  5. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  6. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 18
  7. 2024SparseDM: Toward Sparse Efficient Diffusion Modelscited by 18
  8. 2023Bi-directional Masks for Efficient N:M Sparse Trainingcited by 13
  9. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  10. 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 11
  11. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniquescited by 9
  12. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
20 more
  1. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  2. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  3. 2024Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envisioncited by 5
  4. 2025Neural Architecture Codesign for Fast Physics Applicationscited by 5
  5. 2025Symmetric Pruning of Large Language Modelscited by 3
  6. 2024ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Accelerationcited by 2
  7. 2025EvoP: Robust LLM Inference via Evolutionary Pruningcited by 2
  8. 2023Spatial Re-parameterization for N:M Sparsitycited by 1
  9. 2023SUBP: Soft Uniform Block Pruning for 1xN Sparse CNNs Multithreading Accelerationcited by 1
  10. 2024Torch2Chip: An End-to-end Customizable Deep Neural Network Compression and Deployment Toolkit for Prototype Hardware Accelerator Designcited by 1
  11. 2025One-Cycle Structured Pruning via Stability-Driven Subnetwork Searchcited by 1
  12. 2025MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMscited by 1
  13. 2025From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correctioncited by 1
  14. 2025PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Modelscited by 1
  15. 2023Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Taskscited by 0
  16. 2023Neural Architecture Codesign for Fast Bragg Peak Analysiscited by 0
  17. 2023ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deploymentcited by 0
  18. 2023MaxQ: Multi-Axis Query for N:M Sparsity Networkcited by 0
  19. 2025Extreme Model Compression with Structured Sparsity at Low Precisioncited by 0
  20. 2026Defense Against LLM Backdoors using Critical Neuron Isolation Pruningcited by 0