STAT: Shrinking Transformers After Training
Not on the map. Why?
Abstract (arXiv)
We present STAT: a simple algorithm to prune transformer models without any fine-tuning. STAT eliminates both attention heads and neurons from the network, while preserving accuracy by calculating a correction to the weights of the next layer. Each layer block in the network is compressed using a series of principled matrix factorizations that preserve the network structure. Our entire algorithm takes minutes to compress BERT, and less than three hours to compress models with 7B parameters using a single GPU. Using only several hundred data examples, STAT preserves the output of the network and improves upon existing gradient-free pruning methods. It is even competitive with methods that include significant fine-tuning. We demonstrate our method on both encoder and decoder architectures, including BERT, DistilBERT, and Llama-2 using benchmarks such as GLUE, Squad, WikiText2.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 29 May 2024.
Builds on
47 of 78 references.
References · 47
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Layer Normalizationcited by 5,579
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
35 more
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
- 2019The State of Sparsity in Deep Neural Networkscited by 471
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
- 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
- 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
- 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
- 2019Q8BERT: Quantized 8Bit BERTcited by 296
- 2021I-BERT: Integer-only BERT Quantizationcited by 250
- 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
- 2019Structured Pruning of Large Language Modelscited by 201
- 2021Does Knowledge Distillation Really Work?cited by 187
- 2021Block Pruning For Faster Transformerscited by 176
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
- 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
- 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
- 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
- 2015Data-free parameter pruning for Deep Neural Networkscited by 112
- 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
- 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
- 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
- 2023The LLM Surgeoncited by 36
- 2019Data-Independent Neural Pruning via Coresetscited by 35
- 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
- 2020Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsitycited by 30
- 2020Neuron Merging: Compensating for Pruned Neuronscited by 24
- 2020Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Priorcited by 23
- 2021Post-training deep neural network pruning via layer-wise calibrationcited by 23
- 2021RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Mergingcited by 19
- 2021ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquescited by 14
- 2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
- 2021Model Preserving Compression for Neural Networkscited by 9
- 2021Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architecturescited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.