# Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads

arXiv: https://arxiv.org/abs/2011.03770 · HTML: https://mlsanity.com/paper/2011.03770/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://mlsanity.com/paper/2409.06211/) (2024)
- [Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding](https://mlsanity.com/paper/2402.13720/) (2024)
- [Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling](https://mlsanity.com/paper/2110.03252/) (2021)
- [Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing](https://mlsanity.com/paper/2105.14839/) (2021)
- [CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices](https://mlsanity.com/paper/2307.07705/) (2023)
- [Joint Localization and Activation Editing for Low-Resource Fine-Tuning](https://mlsanity.com/paper/2502.01179/) (2025)
- [Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation](https://mlsanity.com/paper/2104.11928/) (2021)
- [Exploring Attention Map Reuse for Efficient Transformer Neural Networks](https://mlsanity.com/paper/2301.12444/) (2023)
- [Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference](https://mlsanity.com/paper/2407.12893/) (2024)
- [Breaking Down Bias: On The Limits of Generalizable Pruning Strategies](https://mlsanity.com/paper/2502.07771/) (2025)
- [On the Compression of Natural Language Models](https://mlsanity.com/paper/2112.11480/) (2021)
- [DeepCuts: Single-Shot Interpretability based Pruning for BERT](https://mlsanity.com/paper/2212.13392/) (2022)
- [One-Shot Pruning for Fast-adapting Pre-trained Models on Devices](https://mlsanity.com/paper/2307.04365/) (2023)
- [SEVEN: Pruning Transformer Model by Reserving Sentinels](https://mlsanity.com/paper/2403.12688/) (2024)

## Abstract (arXiv)

Deep pre-trained Transformer models have achieved state-of-the-art results over a variety of natural language processing (NLP) tasks. By learning rich language knowledge with millions of parameters, these models are usually overparameterized and significantly increase the computational overhead in applications. It is intuitive to address this issue by model compression. In this work, we propose a method, called Single-Shot Meta-Pruning, to compress deep pre-trained Transformers before fine-tuning. Specifically, we focus on pruning unnecessary attention heads adaptively for different downstream tasks. To measure the informativeness of attention heads, we train our Single-Shot Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the distribution of text representations after pruning. Compared with existing compression methods for pre-trained models, our method can reduce the overhead of both fine-tuning and inference. Experimental results show that our pruner can selectively prune 50% of attention heads with little impact on the performance on downstream tasks and even provide better text representations. The source code will be released in the future.