mlsanityData to 14 Oct 2026

On Multilingual Encoder Language Model Compression for Low-Resource Languages

arXiv 2505.16956 ↗First posted 22 May 2025cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

In this paper, we combine two-step knowledge distillation, structured pruning, truncation, and vocabulary trimming for extremely compressing multilingual encoder-only language models for low-resource languages. Our novel approach systematically combines existing techniques and takes them to the extreme, reducing layer depth, feed-forward hidden size, and intermediate layer embedding size to create significantly smaller monolingual models while retaining essential language-specific knowledge. We achieve compression rates of up to 92% while maintaining competitive performance, with average drops of 2-10% for moderate compression and 8-13% at maximum compression in four downstream tasks, including sentiment analysis, topic classification, named entity recognition, and part-of-speech tagging, across three low-resource languages. Notably, the performance degradation correlates with the amount of language-specific data in the teacher model, with larger datasets resulting in smaller performance losses. Additionally, we conduct ablation studies to identify the best practices for multilingual model compression using these techniques.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 26 of its 43 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

26 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation22 Jan
  2. 2024The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation24 Jun
  3. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models3 Jan

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  11. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  12. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
11 more
  1. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  2. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  3. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  4. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  5. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  6. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32
  7. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  8. 2023MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languagescited by 23
  9. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 20
  10. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  11. 2023Distilling Efficient Language-Specific Models for Cross-Lingual Transfercited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Compression Algorithms for Language Models
  2. 2023A Survey on Model Compression for Large Language Models
  3. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
  4. 2026Break Through the Compression Bottleneck: From Theory to Practice
  5. 2025PocketLLM: Ultimate Compression of Large Language Models via Meta Networks
  6. 2024Efficient Compression of Multitask Multilingual Speech Models
  7. 2025Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks
  8. 2024What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models