mlsanityData to 7 Oct 2026

A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models

arXiv 2305.18239 ↗First posted 26 May 2023cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance. Previous studies have shown that DWT can be effective in the vision domain and natural language processing (NLP) pre-training stage. Specifically, DWT shows promise in practical scenarios, such as enhancing new generation or larger models using pre-trained yet older or smaller models and lacking a resource budget. However, the optimal conditions for using DWT have yet to be fully investigated in NLP pre-training. Therefore, this study examines three key factors to optimize DWT, distinct from those used in the vision domain or traditional knowledge distillation. These factors are: (i) the impact of teacher model quality on DWT effectiveness, (ii) guidelines for adjusting the weighting value for DWT loss, and (iii) the impact of parameter remapping as a student model initialization technique for DWT.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

17 of 26 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
5 more
  1. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  2. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  3. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  4. 2020Fast Neural Network Adaptation via Parameter Remapping and Architecture Searchcited by 16
  5. 2020FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Searchcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 1; 2026: 1.

9 papers.

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  2. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 27
  3. 2024Step Out and Seek Around: On Warm-Start Training with Incremental Datacited by 4
  4. 2024Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoningcited by 4
  5. 2023Co-training and Co-distillation for Quality Improvement and Compression of Language Modelscited by 1
  6. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  7. 2024Induced Model Matching: Restricted Models Help Train Full-Featured Modelscited by 0
  8. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  9. 2025Enhancing Knowledge Transfer in Hyperspectral Image Classification via Cross-scene Knowledge Integrationcited by 0