# TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing

arXiv: https://arxiv.org/abs/2002.12620 · HTML: https://mlsanity.com/paper/2002.12620/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 26 of 26)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models](https://mlsanity.com/paper/2212.10670/) (2022)
- [VkD: Improving Knowledge Distillation using Orthogonal Projections](https://mlsanity.com/paper/2403.06213/) (2024)
- [A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications](https://mlsanity.com/paper/2202.07893/) (2022)
- [AmbiPun: Generating Humorous Puns with Ambiguous Context](https://mlsanity.com/paper/2205.01825/) (2022)
- [ExPUNations: Augmenting Puns with Keywords and Explanations](https://mlsanity.com/paper/2210.13513/) (2022)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [Protein Language Models and Structure Prediction: Connection and Progression](https://mlsanity.com/paper/2211.16742/) (2022)
- [GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model](https://mlsanity.com/paper/2306.06629/) (2023)
- [Knowledge Distillation of Transformer-based Language Models Revisited](https://mlsanity.com/paper/2206.14366/) (2022)
- [TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models](https://mlsanity.com/paper/2203.15996/) (2022)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Progressive Learning without Forgetting](https://mlsanity.com/paper/2211.15215/) (2022)
- [All You Need in Knowledge Distillation Is a Tailored Coordinate System](https://mlsanity.com/paper/2412.09388/) (2024)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Context-Situated Pun Generation](https://mlsanity.com/paper/2210.13522/) (2022)
- [Bridging Fairness and Environmental Sustainability in Natural Language Processing](https://mlsanity.com/paper/2211.04256/) (2022)
- [Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training](https://mlsanity.com/paper/2202.13654/) (2022)
- [MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model](https://mlsanity.com/paper/2304.00717/) (2023)
- [Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.03461/) (2024)
- [Chemical transformer compression for accelerating both training and inference of molecular modeling](https://mlsanity.com/paper/2205.07582/) (2022)
- [Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System](https://mlsanity.com/paper/2107.09625/) (2021)
- [Can depth-adaptive BERT perform better on binary classification tasks](https://mlsanity.com/paper/2111.10951/) (2021)
- [A Small and Fast BERT for Chinese Medical Punctuation Restoration](https://mlsanity.com/paper/2308.12568/) (2023)
- [Distilling Balanced Knowledge from a Biased Teacher](https://mlsanity.com/paper/2506.18496/) (2025)
- [Enriching Knowledge Distillation with Intra-Class Contrastive Learning](https://mlsanity.com/paper/2509.22053/) (2025)

## Abstract (arXiv)

In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters. Our toolkit is available through: http://textbrewer.hfl-rc.com