mlsanityData to 7 Oct 2026

Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures

arXiv 2110.15225 ↗First posted 28 Oct 2021cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent years have seen a growing adoption of Transformer models such as BERT in Natural Language Processing and even in Computer Vision. However, due to their size, there has been limited adoption of such models within resource-constrained computing environments. This paper proposes novel pruning algorithm to compress transformer models by eliminating redundant Attention Heads. We apply the A* search algorithm to obtain a pruned model with strict accuracy guarantees. Our results indicate that the method could eliminate as much as 40% of the attention heads in the BERT transformer model with no loss in accuracy.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

14 of 22 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2019Are Sixteen Heads Really Better than One?cited by 803
  7. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  8. 2018Gender Bias in Coreference Resolutioncited by 438
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  11. 2019Structured Pruning of Large Language Modelscited by 201
  12. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
2 more
  1. 2016"What is Relevant in a Text Document?": An Interpretable Machine Learning Approachcited by 53
  2. 2016Compression of Neural Machine Translation Models via Pruningcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 0; 2026: 2.

6 papers.

  1. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  2. 2023What is Lost in Knowledge Distillation?cited by 2
  3. 2024STAT: Shrinking Transformers After Trainingcited by 2
  4. 2024Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representationcited by 1
  5. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1
  6. 2026Adaptive Head Budgeting for Efficient Multi-Head Attentioncited by 1