mlsanityData to 14 Oct 2026

SambaLingo: Teaching Large Language Models New Languages

arXiv 2404.05829 ↗First posted 8 Apr 2024cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Despite the widespread availability of LLMs, there remains a substantial gap in their capabilities and availability across diverse languages. One approach to address these issues has been to take an existing pre-trained LLM and continue to train it on new languages. While prior works have experimented with language adaptation, many questions around best practices and methodology have not been covered. In this paper, we present a comprehensive investigation into the adaptation of LLMs to new languages. Our study covers the key components in this process, including vocabulary extension, direct preference optimization and the data scarcity problem for human alignment in low-resource languages. We scale these experiments across 9 languages and 2 parameter scales (7B and 70B). We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing language experts, outperforming all prior published baselines. Additionally, all evaluation code and checkpoints are made public to facilitate future research.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

37 of 61 references.

References · 37

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2023The Falcon Series of Open Language Modelscited by 535
  12. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
25 more
  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  4. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  5. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  6. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  7. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  8. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  9. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  10. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  11. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  12. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  13. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  14. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  15. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  16. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  17. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  18. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 38
  19. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  20. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  21. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  22. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  23. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  24. 2023Training Large Language Models Efficiently with Sparsity and Dataflowcited by 7
  25. 2023Impact of Tokenization on LLaMa Russian Adaptationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 3.

11 papers.

  1. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?cited by 41
  2. 2024MKGL: Mastery of a Three-Word Languagecited by 17
  3. 2024Large Language Models Are Cross-Lingual Knowledge-Free Reasonerscited by 16
  4. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  5. 2024Introducing cosmosGPT: Monolingual Training for Turkish Language Modelscited by 8
  6. 2024Open Llama2 Model for the Lithuanian Languagecited by 2
  7. 2025Mangosteen: An Open Thai Corpus for Language Model Pretrainingcited by 2
  8. 2025Are the LLMs Capable of Maintaining at Least the Language Genus?cited by 1
  9. 2026Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructurecited by 1
  10. 2026Domain Specific Specialization in Low-Resource Settings: The Efficacy of Offline Response-Based Knowledge Distillation in Large Language Modelscited by 0
  11. 2026Made in Hungary: Comments on the performance of generative language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  2. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages
  3. 2025NAACL2025 Tutorial: Adaptation of Large Language Models
  4. 2023Extrapolating Large Language Models to Non-English by Aligning Languages
  5. 2024Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque
  6. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
  7. 2024BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment
  8. 2024Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean