mlsanityData to 7 Oct 2026

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages

arXiv 2410.18836 ↗First posted 24 Oct 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script - Ukrainian, Arabic, and Georgian. Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

27 of 51 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  12. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
15 more
  1. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  2. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  3. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  4. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  5. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  6. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  7. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  8. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  9. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  10. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  11. 2023LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Languagecited by 24
  12. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  13. 2023Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Setcited by 12
  14. 2022Lisan: Yemeni, Iraqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotationscited by 9
  15. 2024Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-trainingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  2. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 4
  3. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  4. 2026Writing-System-Level Tokenizer Adaptation for Byte-Level BPEcited by 1
  5. 2025Mirror, Mirror on the Wall -- Which is the Best Model of Them All?cited by 0