mlsanityData to 7 Oct 2026

Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation

arXiv 2305.09651 ↗First posted 16 May 2023cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledge transfer. In order to enhance the guidance of the teacher training process, we introduce the concept of distillation influence to determine the impact of distillation from each training sample on the student's generalization ability. In this paper, we propose Learning Good Teacher Matters (LGTM), an efficient training technique for incorporating distillation influence into the teacher's learning process. By prioritizing samples that are likely to enhance the student's generalization ability, our LGTM outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

40 of 50 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
28 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2017Deep Mutual Learningcited by 621
  4. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  5. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  6. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  7. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  8. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  10. 2019On the Efficacy of Knowledge Distillationcited by 377
  11. 2020Meta Pseudo Labelscited by 372
  12. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 279
  13. 2021Curriculum Learning: A Surveycited by 257
  14. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  15. 2022Knowledge Distillation from A Stronger Teachercited by 204
  16. 2021Does Knowledge Distillation Really Work?cited by 187
  17. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  18. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  19. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  20. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  21. 2019Data Cleansing for Models Trained with SGDcited by 67
  22. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  23. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  24. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  25. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  26. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  27. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  28. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selectioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 2.

8 papers.

  1. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  2. 2023Democratizing Reasoning Ability: Tailored Learning from Large Language Modelcited by 13
  3. 2024Reliable Model Watermarking: Defending Against Theft without Compromising on Evasioncited by 9
  4. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  5. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  6. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  7. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  8. 2026Dynamic Influence-Weighted Distillation for Single-IMU Activity Recognitioncited by 0