mlsanityData to 7 Oct 2026

Targeted Multilingual Adaptation for Low-resource Language Families

arXiv 2405.12413 ↗First posted 20 May 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

The "massively-multilingual" training of multilingual models is known to limit their utility in any one language, and they perform particularly poorly on low-resource languages. However, there is evidence that low-resource languages can benefit from targeted multilinguality, where the model is trained on closely related languages. To test this approach more rigorously, we systematically study best practices for adapting a pre-trained model to a language family. Focusing on the Uralic family as a test case, we adapt XLM-R under various configurations to model 15 languages; we then evaluate the performance of each experimental setting on two downstream tasks and 11 evaluation languages. Our adapted models significantly outperform mono- and multilingual baselines. Furthermore, a regression analysis of hyperparameter effects reveals that adapted vocabulary size is relatively unimportant for low-resource languages, and that low-resource languages can be aggressively up-sampled during training at little detriment to performance in high-resource languages. These results introduce new best practices for performing language adaptation in a targeted setting.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

34 of 54 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  10. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  12. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
22 more
  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  3. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  5. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  6. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  7. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  8. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  9. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  10. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  11. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  12. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  13. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  14. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  15. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  16. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  17. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  18. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  19. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  20. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  21. 2022Toward More Meaningful Resources for Lower-resourced Languagescited by 14
  22. 2021Evaluating Transferability of BERT Models on Uralic Languagescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Reviewcited by 8
  2. 2025Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiencycited by 1
  3. 2026Developing an English-Efik Corpus and Machine Translation System for Digitization Inclusioncited by 1
  4. 2025Teaching a Language Model to Speak the Language of Toolscited by 0
  5. 2026Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluationcited by 0
  6. 2026BhashaSetu: Cross-Lingual Knowledge Transfer from High-Resource to Extreme Low-Resource Languagescited by 0
  7. 2026LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptationcited by 0
  8. 2026Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptationcited by 0
  9. 2026Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nungcited by 0