Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Not on the map. Why?
Abstract (arXiv)
Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, HIES (Head Importance-Entropy Score), which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 61 of its 81 references that are in the map. First posted on arXiv 10 Oct 2025.
Builds on
61 of 81 references.
References · 61
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Visual Instruction Tuningcited by 7,869
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
38 more
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
- 2019Are Sixteen Heads Really Better than One?cited by 816
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
- 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
- 2019Q8BERT: Quantized 8Bit BERTcited by 300
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
- 2024A Survey on Efficient Inference for Large Language Modelscited by 205
- 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
- 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
- 2021Block Pruning For Faster Transformerscited by 177
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
- 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
- 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
- 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
- 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
- 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
- 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
- 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
- 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
- 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
- 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
- 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
- 2020Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domainscited by 33
- 2020Model-Predictive Control via Cross-Entropy and Gradient-Based Optimizationcited by 32
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.