mlsanityData to 14 Oct 2026

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

arXiv 2106.01023 ↗First posted 2 Jun 2021cs.CLcited by 40

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Pre-trained language models (PLMs) achieve great success in NLP. However, their huge model sizes hinder their applications in many practical systems. Knowledge distillation is a popular technique to compress PLMs, which learns a small student model from a large teacher PLM. However, the knowledge learned from a single teacher may be limited and even biased, resulting in low-quality student model. In this paper, we propose a multi-teacher knowledge distillation framework named MT-BERT for pre-trained language model compression, which can train high-quality student model from multiple teacher PLMs. In MT-BERT we design a multi-teacher co-finetuning method to jointly finetune multiple teacher PLMs in downstream tasks with shared pooling and prediction layers to align their output space for better collaborative teaching. In addition, we propose a multi-teacher hidden loss and a multi-teacher distillation loss to transfer the useful knowledge in both hidden states and soft labels from multiple teacher PLMs to the student model. Experiments on three benchmark datasets validate the effectiveness of MT-BERT in compressing PLMs.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 18 of its 27 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

18 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application9 Feb

Other references · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  12. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
5 more
  1. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  2. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 72
  3. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 63
  4. 2020Investigating Gender Bias in BERTcited by 40
  5. 2020Improving Attention Mechanism with Query-Value Interactioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 9; 2024: 13; 2025: 5; 2026: 5.

40 papers.

Sort:

By area:

  1. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  2. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  3. 2022Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillationcited by 62
  4. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  5. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  6. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  7. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  8. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  9. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  10. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teacherscited by 13
  11. 2023Distillation from Heterogeneous Models for Top-K Recommendationcited by 13
  12. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 12
28 more
  1. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
  2. 2024NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Modelscited by 7
  3. 2026Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillationcited by 7
  4. 2023Ensemble Distillation for Unsupervised Constituency Parsingcited by 6
  5. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  6. 2022Unified and Effective Ensemble Knowledge Distillationcited by 5
  7. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  8. 2024Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learningcited by 3
  9. 2021Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Modelscited by 2
  10. 2024MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLUcited by 2
  11. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainabilitycited by 2
  12. 2025Matryoshka Model Learning for Improved Elastic Student Modelscited by 2
  13. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  14. 2022From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Modelscited by 1
  15. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Expertscited by 1
  16. 2024EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translationcited by 1
  17. 2024Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detectioncited by 1
  18. 20243M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health Detectioncited by 1
  19. 2026When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecastingcited by 1
  20. 2021Can depth-adaptive BERT perform better on binary classification taskscited by 0
  21. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  22. 20243M: Multi-modal Multi-task Multi-teacher Learning for Game Event Detectioncited by 0
  23. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  24. 2025Collaborative Unlabeled Data Optimizationcited by 0
  25. 2025TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantizationcited by 0
  26. 2025GLUE: Gradient-free Learning to Unify Expertscited by 0
  27. 2026Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillationcited by 0
  28. 2026EventLens: Event-Structure Reinforcement Learning for Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
  2. 2025Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment
  3. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  4. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
  5. 2022SKDBERT: Compressing BERT via Stochastic Knowledge Distillation
  6. 2021Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
  7. 2022You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models
  8. 2021RefBERT: Compressing BERT by Referencing to Pre-computed Representations