mlsanityData to 14 Oct 2026

TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing

arXiv 2002.12620 ↗First posted 28 Feb 2020cs.CL cs.LG cs.NEcited by 26

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters. Our toolkit is available through: http://textbrewer.hfl-rc.com

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 12; 2023: 3; 2024: 4; 2025: 3.

26 papers.

Sort:

By area:

  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 37
  3. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  4. 2022A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applicationscited by 23
  5. 2022AmbiPun: Generating Humorous Puns with Ambiguous Contextcited by 21
  6. 2022ExPUNations: Augmenting Puns with Keywords and Explanationscited by 21
  7. 2021Information Theoretic Representation Distillationcited by 19
  8. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  9. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  10. 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11
  11. 2022TextPruner: A Model Pruning Toolkit for Pre-Trained Language Modelscited by 10
  12. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
14 more
  1. 2022Progressive Learning without Forgettingcited by 7
  2. 2024All You Need in Knowledge Distillation Is a Tailored Coordinate Systemcited by 6
  3. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  4. 2022Context-Situated Pun Generationcited by 5
  5. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  6. 2022Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Trainingcited by 3
  7. 2023MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Modelcited by 3
  8. 2024Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generationcited by 2
  9. 2022Chemical transformer compression for accelerating both training and inference of molecular modelingcited by 1
  10. 2021Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue Systemcited by 0
  11. 2021Can depth-adaptive BERT perform better on binary classification taskscited by 0
  12. 2023A Small and Fast BERT for Chinese Medical Punctuation Restorationcited by 0
  13. 2025Distilling Balanced Knowledge from a Biased Teachercited by 0
  14. 2025Enriching Knowledge Distillation with Intra-Class Contrastive Learningcited by 0