mlsanityData to 14 Oct 2026

GroupMask: Layer-Adaptive Group-wise Sparsity for Semi-Structured LLM Pruning

arXiv 2609.33977 ↗First posted 27 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Semi-structured pruning compresses large language models (LLMs) while keeping a regular sparse structure, but the prevailing N:M pattern fixes the same local sparsity ratio in every layer. Layer-adaptive sparsity allocation improves unstructured pruning, yet it has been reported to be less effective under N:M sparsity, leaving open whether adaptive allocation is of limited value for semi-structured pruning in general or only under the fine-grained N:M pattern. We examine this question with group-level sparsity, which partitions each weight matrix into regular groups, retains or prunes each group as a whole, and allows each layer's sparsity ratio to vary under a global budget. We propose GroupMask, which generates the group selectors of all layers with a lightweight hypernetwork, relaxes them with a Gumbel-Sigmoid parameterization and a straight-through estimator, and learns them through sparsity-budget regularization and self-distillation while keeping the pretrained weights frozen. On LLaMA-2-7B at 50% sparsity with the same 1×256 group size, learned layer-adaptive allocation reduces WikiText-2 perplexity from 10.02 to 8.30 and raises the average zero-shot accuracy from 0.455 to 0.496 relative to a uniform per-layer ratio. GroupMask obtains the lowest WikiText-2 perplexity on LLaMA-2-7B and the highest average zero-shot accuracy with Alpaca calibration among the evaluated baselines on five LLaMA and Qwen models. Our code is available at https://github.com/ZhengaoLi/GroupMask.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

36 of 47 references.

References · 36

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
24 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  8. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  9. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  10. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  11. 2021Accelerating Sparse Deep Neural Networkscited by 223
  12. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  13. 2021Block Pruning For Faster Transformerscited by 177
  14. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  15. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  16. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  17. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  18. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  19. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  20. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  21. 2022Training Recipe for N:M Structured Sparsity with Decaying Pruning Maskcited by 11
  22. 2025ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMscited by 11
  23. 2019Differentiable Architecture Search with Ensemble Gumbel-Softmaxcited by 6
  24. 2025PATCH: Learnable Tile-level Hybrid Sparsity for LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
  2. 2025SPAP: Structured Pruning via Alternating Optimization and Penalty Methods
  3. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
  4. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Training
  5. 2026When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs
  6. 2025ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
  7. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
  8. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient