mlsanityData to 7 Oct 2026

The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation

arXiv 2406.16524 ↗First posted 24 Jun 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) has proven to be a successful strategy to improve the performance of smaller models in many NLP tasks. However, most of the work in KD only explores monolingual scenarios. In this paper, we investigate the value of KD in multilingual settings. We find the significance of KD and model initialization by analyzing how well the student model acquires multilingual knowledge from the teacher model. Our proposed method emphasizes copying the teacher model's weights directly to the student model to enhance initialization. Our findings show that model initialization using copy-weight from the fine-tuned teacher contributes the most compared to the distillation process itself across various multilingual settings. Furthermore, we demonstrate that efficient weight initialization preserves multilingual capabilities even in low-resource scenarios.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

19 of 27 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
7 more
  1. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  2. 2015All you need is a good initcited by 124
  3. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  4. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  5. 2020Structure-Level Knowledge Distillation For Multilingual Sequence Labelingcited by 13
  6. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  7. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  2. 2025Extracting General-use Transformers for Low-resource Languages via Knowledge Distillationcited by 1
  3. 2025On Multilingual Encoder Language Model Compression for Low-Resource Languagescited by 0
  4. 2026Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmarkcited by 0