Data-Free Pruning of Self-Attention Layers in LLMs
Not on the map. Why?
Abstract (arXiv)
Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training, some deep attention layers learn to mute their own contribution, leaving the residual stream and the MLP to carry the representation. We propose Gate-Norm, a one-shot, weight-only criterion that ranks attention sublayers by query-key coupling and removes the least coupled ones, requiring no calibration data, no forward passes, no fine-tuning, and no specialized kernels. On 40-layer, 13B-parameter LLaMA models, Gate-Norm prunes the model in under a second. Pruning 8-16 attention sublayers yields up to 1.30× higher inference throughput while keeping average zero-shot accuracy within 1.5 percentage points of the unpruned baseline across BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy/Challenge, and OpenBookQA. Across these settings, Gate-Norm matches data-driven pruning methods in accuracy while being ~ 1000× faster to score layers, enabling practical, data-free compression of LLMs.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 3 Dec 2025.
Builds on
21 of 28 references.
References · 21
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
- 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
9 more
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
- 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
- 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
- 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
- 2024What Matters in Transformers? Not All Attention is Neededcited by 83
- 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
- 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
- 2025Mosaic: Composite Projection Pruning for Resource-efficient LLMscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.