mlsanityData to 7 Oct 2026

Dynamic Knowledge Distillation for Pre-trained Language Models

arXiv 2109.11295 ↗First posted 23 Sep 2021cs.CL cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-trained language models. However, existing methods conduct KD statically, e.g., the student model aligns its output distribution to that of a selected teacher model on the pre-defined training dataset. In this paper, we explore whether a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency, regarding the student performance and learning efficiency. We explore the dynamical adjustments on three aspects: teacher model adoption, data selection, and KD objective adaptation. Experimental results show that (1) proper selection of teacher model can boost the performance of student model; (2) conducting KD with 10% informative instances achieves comparable performance while greatly accelerates the training; (3) the student performance can be boosted by adjusting the supervision contribution of different alignment objective. We find dynamic knowledge distillation is promising and provide discussions on potential future directions towards more efficient KD methods. Our code is available at https://github.com/lancopku/DynamicKD.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 23 Sep 2021.

Builds on

24 of 44 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
12 more
  1. 2018Neural Network Acceptability Judgmentscited by 952
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  3. 2019Green AIcited by 620
  4. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2020Calibration of Pre-trained Transformerscited by 307
  7. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  8. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  9. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  10. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  11. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 39
  12. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 10; 2024: 9; 2025: 5; 2026: 3.

38 papers.

Sort:

By area:

  1. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  2. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  3. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  4. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  5. 2024Divide-or-Conquer? Which Part Should You Distill Your LLM?cited by 17
  6. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  7. 2022Dynamic Contrastive Distillation for Image-Text Retrievalcited by 14
  8. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  9. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selectioncited by 12
  10. 2022Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solverscited by 9
  11. 2023Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillationcited by 8
  12. 2023Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adaptercited by 8
26 more
  1. 2022Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategycited by 7
  2. 2024Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Datacited by 7
  3. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  4. 2022MiniDisc: Minimal Distillation Schedule for Language Model Compressioncited by 5
  5. 2022Gradient Knowledge Distillation for Pre-trained Language Modelscited by 5
  6. 2023Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Modelscited by 5
  7. 2024Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translationcited by 5
  8. 2024Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Modelscited by 5
  9. 2025Collaborative Distillation Strategies for Parameter-Efficient Language Model Deploymentcited by 5
  10. 2023ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalizationcited by 3
  11. 2023Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?cited by 3
  12. 2021Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Modelscited by 2
  13. 2022Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Predictioncited by 2
  14. 2025Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizercited by 2
  15. 2022Efficient Sub-structured Knowledge Distillationcited by 1
  16. 2022From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Modelscited by 1
  17. 2023Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attentioncited by 1
  18. 2024GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillationcited by 1
  19. 2026Decoder-Free Distillation for Quantized Image Restorationcited by 1
  20. 2022Hint-dynamic Knowledge Distillationcited by 0
  21. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  22. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  23. 2025A Diversity-Enhanced Knowledge Distillation Model for Practical Math Word Problem Solvingcited by 0
  24. 2025Membership Inference Attack against Large Language Model-based Recommendation Systems: A New Distillation-based Paradigmcited by 0
  25. 2026ReAD: Reinforcement-Guided Capability Distillation for Large Language Modelscited by 0
  26. 2026DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillationcited by 0