mlsanityData to 14 Oct 2026

You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models

arXiv 2210.07135 ↗First posted 13 Oct 2022cs.CLcited by 6

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacher-student knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers' knowledge into a single multilingual student. Our method outperforms standard training methods in low-resource languages and retrains performance on high-resource languages while using the same amount of data. If applied widely, our approach can increase the representation of low-resource languages in NLP systems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 21 of its 27 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

21 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages25 May

Other references · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  12. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
8 more
  1. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  2. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  3. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  4. 2020Balancing Training for Multilingual Neural Machine Translationcited by 72
  5. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  6. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  7. 2020Knowledge Distillation for Multilingual Unsupervised Neural Machine Translationcited by 19
  8. 2020Parsing with Multilingual BERT, a Small Corpus, and a Small Treebankcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 1; 2026: 1.

6 papers.

  1. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
  2. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  3. 2023A Representative Study on Human Detection of Artificially Generated Media Across Countriescited by 22
  4. 2023PuoBERTa: Training and evaluation of a curated language model for Setswanacited by 6
  5. 2025Optimal word order for non-causal text generation with Large Language Models: the Spanish casecited by 2
  6. 2026An In-Vitro Study on Cross-Lingual Generalization in Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021MergeDistill: Merging Pre-trained Language Models using Distillation
  2. 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
  3. 2024The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation
  4. 2023AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference
  5. 2024X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions
  6. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  7. 2025Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
  8. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers