mlsanityData to 7 Oct 2026

Exploring Design Choices for Building Language-Specific LLMs

arXiv 2406.14670 ↗First posted 20 Jun 2024cs.CL cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Despite rapid progress in large language models (LLMs), their performance on a vast majority of languages remains unsatisfactory. In this paper, we study building language-specific LLMs by adapting monolingual and multilingual LLMs. We conduct systematic experiments on how design choices (base model selection, vocabulary extension, and continued pretraining) impact the adapted LLM, both in terms of efficiency (how many tokens are needed to encode the same amount of information) and end task performance. We find that (1) the initial performance of LLM does not always correlate with the final performance after the adaptation. Adapting an English-centric models can yield better results than adapting multilingual models despite their worse initial performance on low-resource languages. (2) Efficiency can easily improved with simple vocabulary extension and continued pretraining in most LLMs we study, and (3) The optimal adaptation method (choice of the base model, new vocabulary size, training data, initialization strategy) is highly language-dependent, and the simplest embedding initialization works well across various experimental settings. Together, our work lays foundations on efficiently building language-specific LLMs by adapting existing LLMs.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

30 of 46 references.

References · 30

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  9. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  12. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
18 more
  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  3. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  4. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  5. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  6. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  7. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  8. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  9. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  10. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  11. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  12. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  13. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  14. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  15. 2022Cedille: A large autoregressive French language modelcited by 20
  16. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  17. 2024RakutenAI-7B: Extending Large Language Models for Japanesecited by 13
  18. 2023Impact of Tokenization on LLaMa Russian Adaptationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 6.

22 papers.

Sort:

  1. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  2. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  3. 2025TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarkingcited by 9
  4. 2024Efficient Continual Pre-training of LLMs for Low-resource Languagescited by 8
  5. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  6. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  7. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  8. 2024From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languagescited by 5
  9. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  10. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  11. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  12. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
10 more
  1. 2025Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estoniancited by 3
  2. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  3. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 2
  4. 2025Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performancecited by 1
  5. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  6. 2026MultiHashFormer: Hash-based Generative Language Modelscited by 1
  7. 2025Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languagescited by 0
  8. 2026Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Trainingcited by 0
  9. 2026From Zero to Hero: An Open LLM Ecosystem for Armeniancited by 0
  10. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0