Pruning Pre-trained Language Models with Principled Importance and Self-regularization
Not on the map. Why?
Abstract (arXiv)
Iterative pruning is one of the most effective compression methods for pre-trained language models. We discovered that finding the optimal pruning decision is an equality-constrained 0-1 Integer Linear Programming problem. The solution to this optimization problem leads to a principled importance criterion which we use to rank parameters during iterative model pruning. To mitigate the poor generalization at high sparsity levels, we propose a self-regularization scheme where model prediction is regularized by the latest checkpoint with increasing sparsity throughout pruning. Our experiments on natural language understanding, question-answering, named entity recognition, and data-to-text generation with various Transformer-based PLMs show the effectiveness of the approach at various sparsity levels.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 21 May 2023.
Builds on
22 of 35 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
10 more
- 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
- 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
- 2015Unifying distillation and privileged informationcited by 233
- 2020DART: Open-Domain Structured Data Record to Text Generationcited by 145
- 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
- 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
- 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
- 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.