# Block Pruning For Faster Transformers

arXiv: https://arxiv.org/abs/2109.04838 · HTML: https://mlsanity.com/paper/2109.04838/

## Key facts

- Cited by 177 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Distilling Task-Specific Knowledge from BERT into Simple Neural Networks](https://mlsanity.com/paper/1903.12136/) (2019)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [Pre-trained Summarization Distillation](https://mlsanity.com/paper/2010.13002/) (2020)
- [Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers](https://mlsanity.com/paper/2002.11794/) (2020)
- [LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression](https://mlsanity.com/paper/2004.04124/) (2020)
- [Compression of Neural Machine Translation Models via Pruning](https://mlsanity.com/paper/1606.09274/) (2016)
- [FastFormers: Highly Efficient Transformer Models for Natural Language Understanding](https://mlsanity.com/paper/2010.13382/) (2020)

## Built on by (most cited first, 30 of 177)

- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models](https://mlsanity.com/paper/2203.07259/) (2022)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://mlsanity.com/paper/2306.11222/) (2023)
- [The Efficiency Misnomer](https://mlsanity.com/paper/2110.12894/) (2021)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [MatFormer: Nested Transformer for Elastic Inference](https://mlsanity.com/paper/2310.07707/) (2023)
- [Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://mlsanity.com/paper/2402.02834/) (2024)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models](https://mlsanity.com/paper/2112.00029/) (2021)
- [Prune Once for All: Sparse Pre-Trained Language Models](https://mlsanity.com/paper/2111.05754/) (2021)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- [EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning](https://mlsanity.com/paper/2210.07795/) (2022)
- [PuMer: Pruning and Merging Tokens for Efficient Vision Language Models](https://mlsanity.com/paper/2305.17530/) (2023)
- [ZipLM: Inference-Aware Structured Pruning of Language Models](https://mlsanity.com/paper/2302.04089/) (2023)
- [SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning](https://mlsanity.com/paper/2207.04606/) (2022)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes](https://mlsanity.com/paper/2402.05406/) (2024)
- [The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://mlsanity.com/paper/2306.03805/) (2023)

## Abstract (arXiv)

Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models. Pruning methods have proven to be an effective way of reducing model size, whereas distillation methods are proven for speeding up inference. We introduce a block pruning approach targeting both small and fast models. Our approach extends structured methods by considering blocks of any size and integrates this structure into the movement pruning paradigm for fine-tuning. We find that this approach learns to prune out full components of the underlying model, such as attention heads. Experiments consider classification and generation tasks, yielding among other results a pruned model that is a 2.4x faster, 74% smaller BERT on SQuAD v1, with a 1% drop on F1, competitive both with distilled models in speed and pruned models in size.