mlsanityData to 14 Oct 2026

Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation

arXiv 2501.12660 ↗First posted 22 Jan 2025cs.CLcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 17 of its 21 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

17 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation24 Jun

Other references · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  9. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  10. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  11. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  12. 2021Improving Large-scale Language Models and Resources for Filipinocited by 20
4 more
  1. 2020Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasetscited by 10
  2. 2023Developing a Named Entity Recognition Dataset for Tagalogcited by 7
  3. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6
  4. 2023Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representationscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025On Multilingual Encoder Language Model Compression for Low-Resource Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models
  2. 2023Letz Translate: Low-Resource Machine Translation for Luxembourgish
  3. 2022Knowledge Distillation of Transformer-based Language Models Revisited
  4. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models
  5. 2023An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models
  6. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees
  7. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  8. 2024A Survey on Symbolic Knowledge Distillation of Large Language Models