mlsanityData to 7 Oct 2026

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

arXiv 1905.09418 ↗First posted 23 May 2019cs.CLcited by 844

Not on the map. Why?

Cited by 45 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by individual attention heads in the encoder to the overall performance of the model and analyze the roles played by them. We find that the most important and confident heads play consistent and often linguistically-interpretable roles. When pruning heads using a method based on stochastic gates and a differentiable relaxation of the L0 penalty, we observe that specialized heads are last to be pruned. Our novel pruning method removes the vast majority of heads without seriously affecting performance. For example, on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads results in a drop of only 0.15 BLEU.

Key facts. Cited by 844 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 123; 2022: 109; 2023: 129; 2024: 137; 2025: 150; 2026: 196.

844 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2022In-context Learning and Induction Headscited by 930
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  11. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  12. 2021Efficient Attentions for Long Document Summarizationcited by 342
38 more
  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  4. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  5. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  6. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  8. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  9. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  10. 2023Continual Pre-training of Language Modelscited by 194
  11. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  12. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  13. 2021Block Pruning For Faster Transformerscited by 176
  14. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  15. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  16. 2023Speculative Decoding with Big Little Decodercited by 163
  17. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  18. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  19. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  20. 2021Learned Token Pruning for Transformerscited by 138
  21. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  22. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  23. 2022Analyzing Transformers in Embedding Spacecited by 131
  24. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  25. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  26. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  27. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  28. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  29. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  30. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  31. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  32. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  33. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  34. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  35. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  36. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  37. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  38. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89

50 most cited of 844 All 844 (JSON)