mlsanityData to 7 Oct 2026

Block Pruning For Faster Transformers

arXiv 2109.04838 ↗First posted 10 Sep 2021cs.LG cs.CLcited by 176

Not on the map. Why?

Abstract (arXiv)

Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models. Pruning methods have proven to be an effective way of reducing model size, whereas distillation methods are proven for speeding up inference. We introduce a block pruning approach targeting both small and fast models. Our approach extends structured methods by considering blocks of any size and integrates this structure into the movement pruning paradigm for fine-tuning. We find that this approach learns to prune out full components of the underlying model, such as attention heads. Experiments consider classification and generation tasks, yielding among other results a pruned model that is a 2.4x faster, 74% smaller BERT on SQuAD v1, with a 1% drop on F1, competitive both with distilled models in speed and pruned models in size.

Key facts. Cited by 176 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

31 of 43 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
19 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 803
  2. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  3. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  4. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  5. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  6. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  7. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  8. 2016FastText.zip: Compressing text classification modelscited by 331
  9. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  10. 2019Structured Pruning of Large Language Modelscited by 201
  11. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  12. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  13. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  14. 2020Pre-trained Summarization Distillationcited by 70
  15. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  16. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  17. 2016Compression of Neural Machine Translation Models via Pruningcited by 43
  18. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  19. 2015Auto-Sizing Neural Networks: With Applications to n-gram Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 31; 2023: 59; 2024: 48; 2025: 22; 2026: 11.

176 papers.

Sort:

By area:

  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  5. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  6. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  7. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  8. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  9. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  10. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  11. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  12. 2021The Efficiency Misnomercited by 98
38 more
  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  3. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  4. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  5. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  6. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  7. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  8. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  9. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  10. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  11. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  12. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  13. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  14. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  15. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  16. 2024A Survey on Transformer Compressioncited by 44
  17. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  18. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  19. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  20. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  21. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
  22. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  23. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  24. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
  25. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  26. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  27. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  28. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 26
  29. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  30. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  31. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  32. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  33. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  34. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  35. 2022Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruningcited by 20
  36. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  37. 2023LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformerscited by 20
  38. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20

50 most cited of 176 All 176 (JSON)