mlsanityData to 14 Oct 2026

Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches

arXiv 2509.22166 ↗First posted 26 Sep 2025cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

The demand for efficient large language model (LLM) inference has intensified the focus on sparsification techniques. While semi-structured (N:M) pruning is well-established for weights, its application to activation pruning remains underexplored despite its potential for dynamic, input-adaptive compression and reductions in I/O overhead. This work presents a comprehensive analysis of methods for post-training N:M activation pruning in LLMs. Across multiple LLMs, we demonstrate that pruning activations enables superior preservation of generative capabilities compared to weight pruning at equivalent sparsity levels. We evaluate lightweight, plug-and-play error mitigation techniques and pruning criteria, establishing strong hardware-friendly baselines that require minimal calibration. Furthermore, we explore sparsity patterns beyond NVIDIA's standard 2:4, showing that the 16:32 pattern achieves performance nearly on par with unstructured sparsity. However, considering the trade-off between flexibility and hardware implementation complexity, we focus on the 8:16 pattern as a superior candidate. Our findings provide both effective practical methods for activation pruning and a motivation for future hardware to support more flexible sparsity patterns. Our code is available https://anonymous.4open.science/r/Structured-Sparse-Activations-Inference-EC3C/README.md .

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

34 of 46 references.

References · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
22 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  3. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  4. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  6. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  7. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  8. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 49
  9. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  10. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
  11. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 21
  12. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 19
  13. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 18
  14. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  15. 2025Inference economics of language modelscited by 9
  16. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  17. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 6
  18. 2024Post-Training Statistical Calibration for Higher Activation Sparsitycited by 5
  19. 2022QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noisecited by 4
  20. 2025R-Sparse R-CNN: SAR Ship Detection Based on Background-Aware Sparse Learnable Proposalscited by 3
  21. 2025ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruningcited by 2
  22. 2025From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correctioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Adaptive Computation with Conformal Certificationcited by 0