mlsanityData to 7 Oct 2026

Are Sixteen Heads Really Better than One?

arXiv 1905.10650 ↗First posted 25 May 2019cs.CLcited by 803

Not on the map. Why?

Cited by 55 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention is a powerful and ubiquitous mechanism for allowing neural models to focus on particular salient pieces of information by taking their weighted average when making predictions. In particular, multi-headed attention is a driving force behind many recent state-of-the-art NLP models such as Transformer-based MT models and BERT. These models apply multiple attention mechanisms in parallel, with each attention "head" potentially focusing on different parts of the input, which makes it possible to express sophisticated functions beyond the simple weighted average. In this paper we make the surprising observation that even if models have been trained using multiple heads, in practice, a large percentage of attention heads can be removed at test time without significantly impacting performance. In fact, some layers can even be reduced to a single head. We further examine greedy algorithms for pruning down models, and the potential speed, memory efficiency, and accuracy improvements obtainable therefrom. Finally, we analyze the results with respect to which parts of the model are more reliant on having multiple heads, and provide precursory evidence that training dynamics play a role in the gains provided by multi-head attention.

Key facts. Cited by 803 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 138; 2022: 104; 2023: 127; 2024: 120; 2025: 138; 2026: 176.

803 papers.

Sort:

By area:

  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2023Steering Language Models With Activation Engineeringcited by 824
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  7. 2021Transformer in Transformercited by 632
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  9. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2022How Do Vision Transformers Work?cited by 305
38 more
  1. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  2. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  3. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  7. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  8. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  9. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  10. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  11. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  12. 2023Continual Pre-training of Language Modelscited by 194
  13. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  14. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  15. 2021Block Pruning For Faster Transformerscited by 176
  16. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  17. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  18. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  19. 2023Speculative Decoding with Big Little Decodercited by 163
  20. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  21. 2022A Review on Language Models as Knowledge Basescited by 156
  22. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  23. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  24. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  25. 2023Spike-driven Transformercited by 141
  26. 2021Learned Token Pruning for Transformerscited by 138
  27. 2021Towards Robust Vision Transformercited by 137
  28. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  29. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  30. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  31. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  32. 2024Transformers are Multi-State RNNscited by 120
  33. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  34. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  35. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  36. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  37. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  38. 2021Finetuning Pretrained Transformers into RNNscited by 94

50 most cited of 803 All 803 (JSON)