mlsanityData to 7 Oct 2026

A Fast Post-Training Pruning Framework for Transformers

arXiv 2204.09656 ↗First posted 29 Mar 2022cs.CL cs.LGcited by 161

Not on the map. Why?

Abstract (arXiv)

Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high training cost and high complexity to model deployment, making it difficult to use in many practical situations. To address this, we propose a fast post-training pruning framework for Transformers that does not require any retraining. Given a resource constraint and a sample dataset, our framework automatically prunes the Transformer model using structured sparsity methods. To retain high accuracy without retraining, we introduce three novel techniques: (i) a lightweight mask search algorithm that finds which heads and filters to prune based on the Fisher information; (ii) mask rearrangement that complements the search algorithm; and (iii) mask tuning that reconstructs the output activations for each layer. We apply our method to BERT-base and DistilBERT, and we evaluate its effectiveness on GLUE and SQuAD benchmarks. Our framework achieves up to 2.0x reduction in FLOPs and 1.56x speedup in inference latency, while maintaining < 1% loss in accuracy. Importantly, our framework prunes Transformers in less than 3 minutes on a single GPU, which is over two orders of magnitude faster than existing pruning approaches that retrain the models.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Builds on 84 of its 106 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

84 of 106 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  14. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  15. 2018Neural Network Acceptability Judgmentscited by 952
  16. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  17. 2019Are Sixteen Heads Really Better than One?cited by 803
  18. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  19. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  20. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  21. 2019Importance Estimation for Neural Network Pruningcited by 531
  22. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  23. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  24. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  25. 2019The State of Sparsity in Deep Neural Networkscited by 471
  26. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  27. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  28. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  29. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  30. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  31. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  32. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  33. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  34. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  35. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  36. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  37. 2019Structured Pruning of Large Language Modelscited by 201
  38. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 54; 2025: 48; 2026: 23.

161 papers.

Sort:

By area:

  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  5. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  7. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  9. 2023An LLM Compiler for Parallel Function Callingcited by 170
  10. 2023Speculative Decoding with Big Little Decodercited by 163
  11. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  12. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
38 more
  1. 2024AI and Memory Wallcited by 89
  2. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  3. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  4. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  5. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  6. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  7. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  8. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  9. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  10. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 44
  11. 2023Discrete Prompt Compression with Reinforcement Learningcited by 42
  12. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  13. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  14. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  15. 2024Attention-Driven Training-Free Efficiency Enhancement of Diffusion Modelscited by 37
  16. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  17. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  18. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  19. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  20. 2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
  21. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  22. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28
  23. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  24. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  25. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  26. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  27. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  28. 2024Pruning Foundation Models for High Accuracy without Retrainingcited by 19
  29. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  30. 2024OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimizationcited by 17
  31. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  32. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  33. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  34. 2024The Need for Speed: Pruning Transformers with One Recipecited by 15
  35. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  36. 2024Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessmentcited by 13
  37. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  38. 2023TULIP: Transformer for Upsampling of LiDAR Point Cloudscited by 12

50 most cited of 161 All 161 (JSON)