mlsanityData to 14 Oct 2026

Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling

arXiv 2110.03252 ↗First posted 7 Oct 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

While Transformer-based models have shown impressive language modeling performance, the large computation cost is often prohibitive for practical use. Attention head pruning, which removes unnecessary attention heads in the multihead attention, is a promising technique to solve this problem. However, it does not evenly reduce the overall load because the heavy feedforward module is not affected by head pruning. In this paper, we apply layer-wise attention head pruning on All-attention Transformer so that the entire computation and the number of parameters can be reduced proportionally to the number of pruned heads. While the architecture has the potential to fully utilize head pruning, we propose three training methods that are especially helpful to minimize performance degradation and stabilize the pruning process. Our pruned model shows consistently lower perplexity within a comparable parameter size than Transformer-XL on WikiText-103 language modeling benchmark.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

18 of 20 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  8. 2019Are Sixteen Heads Really Better than One?cited by 816
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  11. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  12. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
6 more
  1. 2019Adaptive Attention Span in Transformerscited by 149
  2. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 125
  3. 2019Augmenting Self-attention with Persistent Memorycited by 93
  4. 2019Batch-Shaping for Learning Conditional Channel Gated Networkscited by 42
  5. 2020Scheduled DropHead: A Regularization Method for Transformer Modelscited by 28
  6. 2020Know What You Don't Need: Single-Shot Meta-Pruning for Attention Headscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 4; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 97
  3. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  4. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  5. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  6. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 19
  7. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  8. 2025Efficient Multi-Person Motion Prediction by Lightweight Spatial and Temporal Interactionscited by 4
  9. 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3
  10. 2024STAT: Shrinking Transformers After Trainingcited by 2
  11. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1
  12. 2026Linear Predictability of Attention Heads in Large Language Modelscited by 1
2 more
  1. 2025Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selectioncited by 0
  2. 2026Reg4Pru: Regularisation Through Random Token Routing for Token Pruningcited by 0