# TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2203.15996 · HTML: https://mlsanity.com/paper/2203.15996/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Load What You Need: Smaller Versions of Multilingual BERT](https://mlsanity.com/paper/2010.05609/) (2020)
- [TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing](https://mlsanity.com/paper/2002.12620/) (2020)
- [Structured Pruning of a BERT-based Question Answering Model](https://mlsanity.com/paper/1910.06360/) (2019)
- [Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models](https://mlsanity.com/paper/2108.11574/) (2021)

## Built on by (most cited first, 10 of 10)

- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [What Do Compressed Multilingual Machine Translation Models Forget?](https://mlsanity.com/paper/2205.10828/) (2022)
- [Gradient-based Intra-attention Pruning on Pre-trained Language Models](https://mlsanity.com/paper/2212.07634/) (2022)
- [NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models](https://mlsanity.com/paper/2310.10054/) (2023)
- [MUX-PLMs: Data Multiplexing for High-throughput Language Models](https://mlsanity.com/paper/2302.12441/) (2023)
- [Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models](https://mlsanity.com/paper/2210.13086/) (2022)
- [Large Language Model Pruning](https://mlsanity.com/paper/2406.00030/) (2024)
- [Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruning](https://mlsanity.com/paper/2602.04491/) (2026)
- [Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC](https://mlsanity.com/paper/2306.06345/) (2023)
- [COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens](https://mlsanity.com/paper/2509.06836/) (2025)

## Abstract (arXiv)

Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without the labeled data. Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model.