mlsanityData to 14 Oct 2026

TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

arXiv 2203.15996 ↗First posted 30 Mar 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without the labeled data. Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

19 of 22 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2019Are Sixteen Heads Really Better than One?cited by 816
  12. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
7 more
  1. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  2. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  3. 2021Block Pruning For Faster Transformerscited by 177
  4. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32
  5. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 26
  6. 2019Structured Pruning of a BERT-based Question Answering Modelcited by 16
  7. 2021Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 2; 2025: 1; 2026: 1.

10 papers.

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  2. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  3. 2022Gradient-based Intra-attention Pruning on Pre-trained Language Modelscited by 9
  4. 2023NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Modelscited by 7
  5. 2023MUX-PLMs: Data Multiplexing for High-throughput Language Modelscited by 4
  6. 2022Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Modelscited by 2
  7. 2024Large Language Model Pruningcited by 1
  8. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1
  9. 2023Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTCcited by 0
  10. 2025COMPACT: Common-token Optimized Model Pruning Across Channels and Tokenscited by 0